General Experience

Well, this week has been easier than the previous week. Mostly because I have done extra work during week 3. I guess I will call this week “Crash Course on software development and object-oriented programming using Python.” I have created a general framework for our training file, including initial, basic, and linear algorithm. In addition, my code was mostly modular and functionally structured such that adding new models to test, new algorithms, new learning frameworks, etc. wouldn’t affect the functionability of the code. I have also learned how to use Argparse very well. I think I will use it in my upcoming projects! I have also learned about Invariant Risk Minimization, Adapative Risk Minimization, and DomainBed. My main motivation for reading those papers was trying to understand their experimental framework and how we I use their basis to understand the algorithm developed this summer.

Readings (Litrature, online blogposts, and tutorials)

I have read the Invariant Risk Minimization paper and the Adapative Risk Minimization. To understand the Adaptive Risk Minimization paper, I had to go through an article about Meta-Learning. Also, I have read a blog post about Learning Theory: Empirical Risk Minimization. Finally, I have gone through the DomainBed repo, and looked through their code and skimmed through the different algorithms provided by the repo.

Experiments & Algorithms

I have done minimal new experimental work this week since my task was to refactor the code, make it modular, and make it run without breaking if we modify functions, add models, etc. I have replicated my experiements to assert the program’s functionality, and I have also included a CSV data file for the source and target domains.

Experiment

My main experiment was to replicate the results from the DomainBed or, in other words, learn how to use the DomainBed and change some code of it to manipulate it for our testing and research purposes.

Algorithms

I would say that my main algorithm for the code was making it as general as possible. Basically, all that you need to run the script is something similiar to the following:

python runExperiment.py --model=model_name --source_domains=n --target_doamain=m --algorithm=our_algorithm --out_dir=./results

I will extend our work to include neural network framework. Of course this is not any close to a final script, but it is progress!

Results and Findings

Most of my findings this week were research findings and notes about the litrature I’ve reviewed this week. Here is a quick summary:

• The goal of IRM is to build predictors that generalize out-of-distribution, that is, achieving low error across all the envrionments. To this end, IRM enforces low error and invariance across transfer environments.
• By constructing a colored MNIST where we randomly choose different digits to be either red or green, we are creating suprious correlations that will make a correlational, typical ERM model overfit and use them as basis for distringuishing colors. However, when we apply our IRM method, we find that IRM performs much better on test envrionments than the ERM Algorithm. The main reason for that is that IRM looks for invariant correlation (more specifically, what makes a digit a digit? regardless of the pixel colors, etc.)
• A good meta-learning model should be trained over a variety of learning tasks and optimized for the best performance on a distribution of tasks, including potentially unseen tasks.
• adaptive risk minimization (ARM) proposes the following objective: optimize the model such that it can maximally leverage the unlabeled adaptation phase to handle group distribution shift.
• ARM methods improve robustness and performance across all experiments conducted in the ARM paper. ARM methods consistently increase both worst case and average accuracy compared to all other methods.

Frustrations

I didn’t have a lot of frustrations this week. Partly because - and again repeating it! - I haven’t encountered new knowledge or information. I would mainly stick to the common frustration of not knowing why your code isn’t running as expected even though you have tried the exact same steps on StackOverFlow and Github Discussions!

Plans for Next Week

My plans for next week is to get used to DomainBed more, and start implementing basic ERM algorithms on the framework, adding some functions and modifying some parameters.

Other things worth sharing

Still happy and getting more excited. Now, I am looking for real world applications where I can apply causality. I have been fascinated by how powerful causality can be and how it can answer ethical questions in many ML Algorithms. I actually had a talk in my school about bias and discrimination in AI, and I am very happy about studying it in a mathematical, technical, jargon-y way! Also, I am almost halfway my summer!