the yt project's blog
© 2023 yt project blog
Example HEALpix Contour Rendering Movie
In response to Matt’s post <http://blog.enzotools.org/yt-development-healpix- and-contour-tree> on the HEALpix rendering update, I thought it would be worth posting an example movie. This shows the all-sky rendering of an observer moving from the front face of a simulation through the volume to the back face. The test simulation is 32 Mpc/h on a side with 64^3 root grid cells and up to 4 levels of refinement. At the start it looks like a disc because the entire simulation is in front of the camera and by the end it is all around the sides, indicating the simulation is behind the camera.
yt development - HEALpix and Contour Tree
This week there was not very much yt development. However, a few notes may be of interest. SamS has updated the HEALPix camera to support ordered projections; what this means is that you can now make volume renderings using a standard color transfer function, or even the Planck transfer function, that cover 4pi of the sky. I am still working on integrating a method for creating images easily, but for now the scripts from last week should work.
yt development - All-sky column density calculation
This week I added the ability to calculate all-sky column densities. This functionality uses HEALpix to calculate equally-area regions on the sky and then shoots out rays from a central source to some fixed radius, accumulating values of a field along the way. Although so far I’ve only used it to calculate column densities of “Density” it could be used for other values as well, including all- sky weighted averages of quantities.
yt development - Documentation
As a result of progress in my scientific goals, and the application of recent yt developments to them, I did not make many changes or developments in yt this week. When I did work on yt, I primarily spent time re-organizing the documentation and fixing several errors. I have added an “installation” section, consolidated a few sections, and wrote two new sections on how to make plots and on how to generate derived data products.
yt development - Time series, and more
Not much yt development went on in the last week; I spent some time working with Enzo and driving forward simulation goals, which resulted in some development that directly benefited those simulation goals. However, this fortuitously coincided with work I have been eager to return to for quite some time: namely, time series analysis!
Time Series Analysis The problem with time series analysis in yt has, to this point, been an issue of verbosity and clunkiness.
yt development - 2.0, Cython, and physics module wrapping
This is the second blog entry in the weekly series, with some updates on what took place last week with respect to yt development. One of the more exciting things is the final one, which is the start of what I want to focus on for the next couple months or years: integration of physics modules with analysis code, and then the ultimate inversion of that relationship.
yt-2.0 This week saw the release of yt 2.
yt development - star particle rendering, simple merger trees and documentation
This is the first of a new series of “what’s up with yt” blog posts I’m going to be writing. By keeping this log, I hope that maybe some things that would otherwise get lost in the version control changesets will get brought to greater light. This covers the time period of the first couple weeks in January.
Star Particle Rendering On the mailing list, the question of adding star particles to a volume rendering was raised.
AMR kd-Tree rendering added to yt
After a significant amount of development and restructuring, I have added the AMR kd-Tree rendering framework to yt. There are several posts on this blog about this module already, so I won’t go over all the background information again. Here I’d like to showcase some of the recent successes and capabilities of the volume rendering within yt.
New optimization options:There are a few important additions that have made it possible to render some of the largest AMR simulations we have available.
Enzo 2.0 and Inline yt
Enzo 2.0 has just been released to its new Google Code website. This release features preliminary support for inline Python analysis, using yt.
In the Enzo documentation there’s a brief section on how to use yt for inline analysis. As it stands, many features are not fully functional, but things like phase plots, profiles, derived quantities and slices all work. This functionality is currently untested at large (> 128) processors, but for small runs – particularly debugging runs!
kD-Tree Rendering Improvements
Hi all,
Just sharing a video here that showcases some improvements I’ve made to the kD-tree rendering that will be making its way to yt for the 2.0 release. You can download it render_movie.3gp|here
Just to be clear this is showing the rendering of a cosmology simulation with a 64^3 root grid + 6 AMR levels in real time on 8 processors. The script is run in parallel, with the root processor displaying the results once each frame is finished.
Improvements to Parallelism
The last few days I’ve spent some time looking at how parallelism in yt performs. I conducted two different tests, both of which operated on the 512^3, 7 level ‘Santa Fe Light Cone’ dataset RD0036. This dataset has 5.5e8 total cells and in the neighborhood of 380,000 grids. I ran four different tests: a 1D profile of the entire dataset, a 2D profile of the entire dataset, and projections of both ‘Density’ (requires IO) and ‘Ones’ (doesn’t require IO).
Quad Tree Projections
The current method for projections in yt is based on a home-grown algorithm for calculating grid overlap and joining points. I’ve always been pretty proud of it – it gave good results, and it succeeded at the project-once-make-many-images philosophy that went into its design. Rather than storing a 2D array of pixels, projections and slices in yt store flat arrays of image plane coordinates and cell widths. This means that there’s an additional step of pixelization to create an image, but it also means that arbitrary images can be made from a single projection or slice operation.
How the Merger Tree Sped Up SQLite Database UPDATEs
The Parallel Merger Tree in yt, like most of the code in yt, has a rich history of changes and upgrades. One of the most significant upgrades was a change in the way the SQLite database file is updated during the course of building the merger tree. Briefly, the database contains all the information about the merger tree of the halos, as well as the specifics of each halo, such as the mass, position or bulk velocity.
yt has moved to mercurial!
For about a year and a half now, most of the unstable development of yt has occurred inside a mercurial repo. Mercurial is a distributed version control system, not unlike git or bzr, where each checkout brings with it the entire history of the repository and enables full development. Each individual can commit changes to their own repository, while still accepting changes from others. It also makes it much easier to submit patches upstream.
Figuring Out Stereo Volume Rendering
Last week I was approached by a friend and collaborator to prepare some large volume renderings using the software volume renderer in yt. In the past we’ve successfully made very, very large image renderings using yt – Sam’s even made one at 8192^2, although at extremely high resolution like that sometimes the lack of fidelity in the underlying volume renderer shows up; sometimes even artifacts in the AMR grid boundaries, but that’s less common.
Volume Rendering with a kD-Tree Decomposition
Let’s talk about the progress we’ve made on a kD-Tree decomposition.
Aboout the yt Project
Want to contribute a post? Check out our contributor guide
Check out our documentation here
The yt-project homepage
Our quickstart notebooks on getting started
The yt source repository
yt extension modules
yt has many extension packages to help you in your scientific workflow! Check these out, or create your own.
ytini
ytini is set of tools and tutorials for using yt as a tool inside the 3D visual effects software Houdini or a data pre-processor externally to Houdini.
Trident
Trident is a full-featured tool that projects arbitrary sightlines through astrophysical hydrodynamics simulations for generating mock spectral observations of the IGM and CGM.
pyXSIM
pyXSIM is a Python package for simulating X-ray observations from astrophysical sources.
ytree
Analyze merger tree data from multiple sources. It’s yt for merger trees!
yt_idv
yt_idv is a package for interactive volume rendering with yt! It provides interactive visualization using OpenGL for datasets loaded in yt. It is written to provide both scripting and interactive access.
widgyts
widgyts is a jupyter widgets extension for yt, backed by rust/webassembly to allow for browser-based, interactive exploration of data from yt.
yt_astro_analysis
yt_astro_analysis is the yt extension package for astrophysical analysis.
Make your own!!
Finally, check out our development docs on writing your own yt extensions!
Contributing to the Blog
Are you interested in contributing to the yt blog?
Check out our post on contributing to the blog for a guide!
We welcome contributions from all members of the yt community. Feel free to reach out if you need any help.
the yt data hub
The yt hub at https://girder.hub.yt/ has a ton of resources to check out, whether you have yt installed or not.
The collections host all sorts of data that can be loaded with yt. Some have been used in publications, and others are used as sample frontend data for yt. Maybe there’s data from your simulation software?
The rafts host the yt quickstart notebooks, where you can interact with yt in the browser, without needing to install it locally. Check out some of the other rafts too, like the widgyts release notebooks – a demo of the widgyts yt extension pacakge; or the notebooks from the CCA workshop – a user’s workshop on using yt.