Setting up a structure for long-term projects

10 March 2023

Since I started my astrophotography journey I have tried to take a picture of Orion at least once a year. Sometimes as an overview, sometimes more in detail like the Horsehead or the Great Orion Nebula. This year I wanted to create an image by combining broadband and narrowband data. But as so often either the weather was not good, Orion was not visible from my location or I had other projects to finish.

In mid-February I had the chance to collect some broadband data of the Horsehead and was hoping to get some narrowband data some time later. But now, at the end of winter, the Orion photography season is over and I am stuck with broadband data. The result is shown above.

But now I have a problem: Up to now I have stored my files in individual folders, as shown below:

...
2020-04-19 M51 Whirlpool Galaxy
    D7500
        bias
        darks
        flats
        lights        
2022-03-07 M51,M81,M82
    2022-03-07_NIGHT_1
        Bias
        Dark
        Flat
        Light
            OBJECT_M51
            OBJECT_M81_M82
    2022-03-08_NIGHT_2
        Bias
        Dark
        Flat
        Light
            OBJECT_M51
            OBJECT_M81_M82
    2022-03-09_NIGHT_3
        ...
...

Each short-term project had a folder whose name started with the first observation date in ISO format (YYYY-MM-DD), and the structure below that folder was quite different for each new project. This is fine as long as the project is only going to live for a short period of time, but for projects that are going to last for several months or even years, this structure is simply not feasible: Where should I store my images? When was the last time I worked on this project? I cannot remember the date and therefore the correct folder name.

So when I started my first multi-year project, I had to come up with a structure that better suited my needs. As well as making it easy to find the project, I wanted to make sure I could use the WBPP feature to add a root folder and automatically separate images by night, filter, darks, flats, lights and so on. Here is the structure I thought of:

<TELESCOPE>_<CAMERA>
    <PROJECT-NAME>
        data
            FILTER_<FILTER>
                NIGHT_<YYYY-MM-DD>
                    Flat
                    Light
                        PANEL_<XYZ>

Where <TELESCOPE>_<CAMERA> describes the combination of telescope and camera. This may be a problem later, but at the moment I don't plan to use different cameras for the same project. <PROJECT-NAME> is the name of the project, usually containing the objects photographed. data' will contain the raw data, e.g. the files. This way I can create processing folders as siblings, containing processing at different stages of the project. <FILTER> will describe the filter I used. Possible values are broadband, r, g, b, lum, duo-narrowband, ha, oiii, sii.

Filled with some data, this looks like the following section:

130PDS_ASI294MCpro
    Bode's Galaxy and Cigar Galaxy
        data
            FILTER_broadband
                NIGHT_2023-02-20
                    Flat
                    Light
    Horsehead Nebula and Flame Nebula
        data
            FILTER_broadband
                NIGHT_2023-02-20
                    Flat
                    Light
    Pinwheel Galaxy
        data
            FILTER_broadband
                NIGHT_2023-02-13
                    Flat
                    Light

My library of correction frames for darks and bias are stored in a different hierarchy.

With this new structure, I hope to get very deep images over the next few years, and it allows me to find started projects quickly and easily.