How and why I built a Machine Learning application to detect features in real-time on the Sun's surface

About me

Lecturer and scientist. Interested in computer science, astronomy, and archaeoastronomy. Love a challenge and working on coding problems.

The problem I wanted to solve

I own a solar scope and an astronomy camera. Seeing lots of tutorials for detecting pets, traffic lights, or fish, I decided to see if the same algorithms and models could be applied to detect the Sun's surface features. The problem in my case was the specific setup (ASI ZWO camera on USB2 and the solar scope itself used without a mount) and the little train data I have gathered (I had the scope for several months but the UK weather is not helping so I only had 14 days worth of images).

What is Machine learning to detect features in real-time on the Sun's surface?

After about 2 days of work, I ended up with an application that can detect in real-time from a live stream or from a prerecorded video the most essential features on the Sun's surface: filaments, faculae, prominences, and sunspots.

Tech stack

Software:

• Python3.7
• Tensorflow and Keras
• OpenCV
• ASI ZWO SDK
• LabelImg

Hardware:

• ASI 120MC astronomy camera on USB2.0 connected to a solar telescope
• Laptop without GPU running Windows 10

The process of building a Machine Learning application to detect features in real-time on the Sun's surface

1. Gathering the images and labeling the train/test data using existing tools
2. Installing the required libraries
3. Compiling the model by following online tutorials
4. Writing the detection script by updating existing ones and adapting them to communicate with the specific camera I use
5. Testing the script on live solar images (I saw the Sun briefly yesterday so I took the chance) and on my previously recorded videos.

Challenges I faced

• Integrating the ASI camera with Python as OpenCV does not detect it
• Making the existing scripts work as I had to navigate through a lot of API versions and even change code inside existing library and model scripts
• Long training time on my laptop (no GPU). It took over 4 hours to bring the loss to <0.0xxxx.

Key learnings

• Navigating through different APIs and gluing everything together.
• Integrating ASI cameras with OpenCV.
• Using LabelIMG to annotate my train/test images

Tips and advice

My full story can be found on my blog at https://starscape-experiences.space/index.php/2023/01/05/detecting-solar-features-automatically-using-machine-learning/

Final thoughts and next steps

Good start but it still requires more training data as it misses some obvious features.
The detected features change from frame to frame so it is worthwhile investigating differences in frames.
I plan on integrating this into my Pi4 remote observatory setup or live stream it to a server for remote feature detection.