the yt project's blog
© 2023 yt project blog
New Screencasts
A few of the yt developers have been experimenting with screencasts to show off new features or demonstrate how to do some things. Sam and I have both prepared screencasts on volume rendering and getting started with developing, respectively. Check them out below, and please feel free to leave comments and let us know what you think – not just about the screencasts, but about what they demonstrate, and if you think any of the concepts or routines could be made easier.
Dataset Tracking with yt
In this post I’d like to discuss a bit of work in progress to highlight some exciting new features that we hope to have working in yt sometime soon.
On any machine that runs yt, there is a file created in the users home directory named ~/.yt/parameter_files.csv that yt uses internally to keep track of datasets it has seen. This is just a simple text file containing comma-separated entries with a few pieces of information about datasets, like their location on disk and the last date and time they were ‘seen’ by yt.
Optimizing yt
This last week, following the release of version 2.2 of yt, I spent a bit of time looking at speed improvements. There were several places that the code was unacceptably slow:
1D profiles (as noted in our method paper, even) Ghost-zone generation RAMSES grid data loading The first of these was relatively easy to fix. In the past, 1D profiles (unlike 2D profiles) were calculated using pure-python mechanisms; numpy was used for digitization, then inverse binning was conducted by the numpy ‘where’ command, and these binnings were used to generate the overall histogram.
yt-2.2 Release Announcement
(Please feel encouraged to forward this message to any other interested parties.)
We are proud to announce the release of yt version 2.2. This release includes several new features, bug fixes, and numerous improvements to the code base and documentation. At the new yt homepage, http://yt-project.org/ , an installation script, a cookbook, documentation and a guide to getting involved can be found. We are particularly proud of the new GUI, entitled “Reason,” which allows real-time exploration of datasets, and which can be used (locally or remotely over SSH) with no dependencies other than a web browser.
This blog has moved -- but you don't have to!
In keeping with the project rename we’ve moved this blog from it’s old home at blog.enzotools.org to its new home at blog.yt-project.org. But, we’ve put in a few redirects, and the RSS feed hasn’t moved, so you shouldn’t need to do anything different to get here. We’ve also enabled anonymous commenting, so feel free to comment below. (But, of course, letting us know who you are would certainly help with keeping in touch!
Introducing the yt Hub
I am very excited to announce the yt Hub: http://hub.yt-project.org/.
The yt Hub is a gathering place for all things related to running, analyzing, and presenting astrophysical simulations. Once upon a time, we had the Enzotools Barn, which was a place to share Enzo scripts unrelated to yt, including hosting of those scripts. The Barn was a minor success, but had a very limited scope and a somewhat clunky user experience.
We're now the yt Project!
This is just a brief blog entry – other news is on the way soon! – to let everyone know we’ve updated our domains. In keeping with the change in focus of the yt project, expanding to work with multiple codes and take broader views of analysis and visualization, we have moved from enzotools.org to yt-project.org. Some other exciting changes will be coming from this in the future, too!
For more info, see the mailing list announcement.
A movie of yt in paraview
Jorge posted this video of using yt inside ParaView. It demonstrated multi-res slices, projections, and even volume rendering with an adjustable transfer function. I think it’s pretty incredible. Thanks very much to Jorge and Kitware for their hard work, and it’ll be exciting to see where it goes from here!
A mission statement and a new homepage!
In case you haven’t visited our front page lately, the yt homepage has been redesigned! The old homepage was nice and functional, but it tried to cram too much into too small a place. The new homepage focuses on the three main questions people have when they go to the yt page: ‘How do I get it?', ‘How do I use it?', and ‘How do I participate?’
The new page is focused on some concrete examples, some quantitative analysis, and features a heavy dosage of the yt community – in fact, the redesign came about through a conversation in IRC, where a few of the core developers were speculating about how best to convey how to participate and how strong a community yt has.
ParaView and yt
Thanks to some awesome work by developers at Kitware, yt can now be called from within ParaView!
This relies on a couple things, all of which are pretty exciting moving forward. The idea behind this is to leverage where the two codes have their own strengths and weaknesses, and identify places where they can work productively together. For instance, yt has been designed to provide astrophysical analysis; in this way, it can generate adaptive projections, spectral energy distributions and so on.
Gmaps-ify your data!
A couple of us have been working on a comprehensive notebook web GUI for yt. It’s not ready yet.
BUT! In advance of that, we’ve rolled a portion of that into something called the ‘mapserver’ into the development branch. This is a small, standalone webapp that implements a rendered-on-the-fly google maps interface in yt. To run it, just go into a directory that has some data, and run:
yt mapserver DD0054/DD0054 (where ‘DD0054/DD0054’ is the same thing you’d feed to ‘load’ in a script.
yt development - Quad trees, Tickets and more
It’s been a while since the last Development post — but in that time, some pretty fun things have been going on. These are some of the smaller things, but there are bigger things in store which I’ll write about next time.
QuadTree Projections For a while, there has been the option to use a QuadTree data structure to conduct projections through a simulation volume. This was accessible through the quad_proj attribute, which respected the same interface as the normal proj interface.
yt Logo Contest, and possibly a free mug!
yt is in need of an awesome new logo, which is why we are announcing the first ever new logo contest! So here’s the deal:
We’ll accept entries for the next two weeks. Let’s arbitrarily say Tuesday, May 10th, at 11:59:59 pm EST. If there is more than one entry, we will put it up for a community vote at that time.
Oh, and the most important part – the winner will get a coffee mug with their new logo front and center!
yt 2.1 Release Announcement
We are proud to announce the release of yt version 2.1. This release includes several new features, bug fixes, and numerous improvements to the code base and documentation. At the yt homepage, http://yt.enzotools.org/ , an installation script, a cookbook, documentation and a guide to getting involved can be found.
yt is an analysis and visualization toolkit for Adaptive Mesh Refinement data. yt provides full support for Enzo, Orion, and FLASH codes, with preliminary support for RAMSES, ART, Chombo, CASTRO and MAESTRO codes.
yt development - Treecodes, GUIs, IRC and more!
It’s been nearly a month since the last yt development post; in that time, there’s been quite a bit of development in a couple different areas. This is culminating in a 2.1 release, for which Sam Skillman is release manager, sometime in the next few days.
Streamlines and Treecode SamS has spent some time over the last month developing two types of streamline code. The first integrates a series of streamlines over a selection of the domain, which can then be visualizing using the mplot3d package.
yt development - External Analysis and Simulation Code Support
This last week was the first full week on BitBucket and so far I think it has been quite successful. The new development process is for most of the core developers to maintain personal forks for experimental changes, or longer term changes, and then to commit directly or merge when bug fixes or features are ready to be integrated. The list of forks is easily visible and each individual fork’s divergence from the primary repository can be viewed by clicking on the green arrows.
yt development - BitBucket, Task Queues, and Streamlines
The major changes this week came mostly in the form of administrative shifts. However, SamS did some great work I’m going to hint at (he’ll post a blog entry later) and I started laying the ground work for something I’ve been excited about for a while, an MPI-aware task queue.
BitBucket For the last couple months, yt has been struggling under the constraints of the hg server on its hosting plan.
Example HEALpix Contour Rendering Movie
In response to Matt’s post <http://blog.enzotools.org/yt-development-healpix- and-contour-tree> on the HEALpix rendering update, I thought it would be worth posting an example movie. This shows the all-sky rendering of an observer moving from the front face of a simulation through the volume to the back face. The test simulation is 32 Mpc/h on a side with 64^3 root grid cells and up to 4 levels of refinement. At the start it looks like a disc because the entire simulation is in front of the camera and by the end it is all around the sides, indicating the simulation is behind the camera.
yt development - HEALpix and Contour Tree
This week there was not very much yt development. However, a few notes may be of interest. SamS has updated the HEALPix camera to support ordered projections; what this means is that you can now make volume renderings using a standard color transfer function, or even the Planck transfer function, that cover 4pi of the sky. I am still working on integrating a method for creating images easily, but for now the scripts from last week should work.
yt development - All-sky column density calculation
This week I added the ability to calculate all-sky column densities. This functionality uses HEALpix to calculate equally-area regions on the sky and then shoots out rays from a central source to some fixed radius, accumulating values of a field along the way. Although so far I’ve only used it to calculate column densities of “Density” it could be used for other values as well, including all- sky weighted averages of quantities.
yt development - Documentation
As a result of progress in my scientific goals, and the application of recent yt developments to them, I did not make many changes or developments in yt this week. When I did work on yt, I primarily spent time re-organizing the documentation and fixing several errors. I have added an “installation” section, consolidated a few sections, and wrote two new sections on how to make plots and on how to generate derived data products.
yt development - Time series, and more
Not much yt development went on in the last week; I spent some time working with Enzo and driving forward simulation goals, which resulted in some development that directly benefited those simulation goals. However, this fortuitously coincided with work I have been eager to return to for quite some time: namely, time series analysis!
Time Series Analysis The problem with time series analysis in yt has, to this point, been an issue of verbosity and clunkiness.
yt development - 2.0, Cython, and physics module wrapping
This is the second blog entry in the weekly series, with some updates on what took place last week with respect to yt development. One of the more exciting things is the final one, which is the start of what I want to focus on for the next couple months or years: integration of physics modules with analysis code, and then the ultimate inversion of that relationship.
yt-2.0 This week saw the release of yt 2.
yt development - star particle rendering, simple merger trees and documentation
This is the first of a new series of “what’s up with yt” blog posts I’m going to be writing. By keeping this log, I hope that maybe some things that would otherwise get lost in the version control changesets will get brought to greater light. This covers the time period of the first couple weeks in January.
Star Particle Rendering On the mailing list, the question of adding star particles to a volume rendering was raised.
AMR kd-Tree rendering added to yt
After a significant amount of development and restructuring, I have added the AMR kd-Tree rendering framework to yt. There are several posts on this blog about this module already, so I won’t go over all the background information again. Here I’d like to showcase some of the recent successes and capabilities of the volume rendering within yt.
New optimization options:There are a few important additions that have made it possible to render some of the largest AMR simulations we have available.
Aboout the yt Project
Want to contribute a post? Check out our contributor guide
Check out our documentation here
The yt-project homepage
Our quickstart notebooks on getting started
The yt source repository
yt extension modules
yt has many extension packages to help you in your scientific workflow! Check these out, or create your own.
ytini
ytini is set of tools and tutorials for using yt as a tool inside the 3D visual effects software Houdini or a data pre-processor externally to Houdini.
Trident
Trident is a full-featured tool that projects arbitrary sightlines through astrophysical hydrodynamics simulations for generating mock spectral observations of the IGM and CGM.
pyXSIM
pyXSIM is a Python package for simulating X-ray observations from astrophysical sources.
ytree
Analyze merger tree data from multiple sources. It’s yt for merger trees!
yt_idv
yt_idv is a package for interactive volume rendering with yt! It provides interactive visualization using OpenGL for datasets loaded in yt. It is written to provide both scripting and interactive access.
widgyts
widgyts is a jupyter widgets extension for yt, backed by rust/webassembly to allow for browser-based, interactive exploration of data from yt.
yt_astro_analysis
yt_astro_analysis is the yt extension package for astrophysical analysis.
Make your own!!
Finally, check out our development docs on writing your own yt extensions!
Contributing to the Blog
Are you interested in contributing to the yt blog?
Check out our post on contributing to the blog for a guide!
We welcome contributions from all members of the yt community. Feel free to reach out if you need any help.
the yt data hub
The yt hub at https://girder.hub.yt/ has a ton of resources to check out, whether you have yt installed or not.
The collections host all sorts of data that can be loaded with yt. Some have been used in publications, and others are used as sample frontend data for yt. Maybe there’s data from your simulation software?
The rafts host the yt quickstart notebooks, where you can interact with yt in the browser, without needing to install it locally. Check out some of the other rafts too, like the widgyts release notebooks – a demo of the widgyts yt extension pacakge; or the notebooks from the CCA workshop – a user’s workshop on using yt.