Run the Generalized IOP (GIOP) inversion model in the Level-2 Generatorl2gen OCSSW program on OCI data

Run the Generalized IOP (GIOP) inversion model in the Level-2 Generator`l2gen` OCSSW program on OCI data#

Authors: Anna Windle (NASA, SSAI), Jeremy Werdell (NASA)
Last updated: August 3, 2025

The following notebooks are prerequisites for this tutorial.

Learn with OCI: Data Access
Learn with OCI: Installing and Running OCSSW Command-line Tools
Learn with OCI: Run Level-2 Generator (l2gen) OCSSW program on OCI data

An Earthdata Login account is required to access data from the NASA Earthdata system, including NASA ocean color data.

This notebook was desgined to use cloud-enabled OCSSW programs, which are available in OCSSW tag V2025.2 or higher. Cloud-enabled OCSSW programs can only be run on an AWS EC2 instance, such as CryoCloud.

Summary#

The OceanColor Science SoftWare (OCSSW) repository is the operational data processing software package of NASA’s Ocean Biology Distributed Active Archive Center (OB.DAAC). OCSSW is publically available through the OB.DAAC’s Sea, earth, and atmosphere Data Analysis System (SeaDAS), which provides a complete suite of tools to process, display and analyze ocean color data. SeaDAS is a desktop application that provides GUI access to OCSSW, but command line interfaces (CLI) also exist, which we can use to write processing scripts and notebooks without the desktop application.

In a previous tutorial, we demonstrated how to run the Level-2 Generator (l2gen) program to generate aquatic Level-2 (L2) data products from calibrated top-of-atmosphere (TOA) radiances. In practice, l2gen not only allows generation of aquatic geophysical products, but also the capability to (re)parameterize the algorithms used to generate them. For example, inherent optical properties (IOPs) are routinely produced and distributed as part of NASA standard L2 and Level-3 (L3) IOP suites, inclusive of spectral absorption and backscattering coefficients and their subcomponents. These standard products are generated using the default configuration of the Generalized IOP (GIOP) framework (Werdell et al., 2013). Using l2gen, you can select options to generate additional IOPs or reconfigure the parameterization of GIOP.

This tutorial will demonstrate how to process PACE OCI L1B data through l2gen to retrieve the standard L2 IOP data suite. This tutorial wil also demonstrate how to modify the operation of l2gen confifurations based on your research needs.

Learning Objectives#

At the end of this notebook you will know:

How to navigate and open files related to the GIOP within OCSSW directory
How to process L1B data to retreive L2 IOPs with l2gen
Modifications you can make to l2gen to obtain different IOP products

Contents#

Setup
Search and access L1B Data
Run l2gen with default GIOP configurations
Plot L2 IOP data products
Run GIOP with modifications to the configurations

1. Setup#

Begin by importing all of the packages used in this notebook. If your kernel uses an environment defined following the guidance on the tutorials page, then the imports will be successful.

import csv
import os
from pathlib import Path

import earthaccess
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import xarray as xr

Next, we’ll set up the OCSSW programs.

OCSSW programs are system commands, typically called in a Bash shell. We will employ system shell access available in the IPython kernel to launch system commands as a subprocess using the ! prefix. In the specific case of OCSSW programs, a suite of required environment variables must be set by first executing source $OCSSWROOT/OCSSW_bash.env in the same subprocess.

Every time we use ! to invoke an OCSSW program, we must also evaluate the OCSSW_bash.env environment file shipped with OCSSW. Each ! initiated subprocess is distinct, and the environment configuration is discarded after the command is finished. Let’s get prepared by reading the path to the OCSSW installation from the OCSSWROOT environment variable (assuming it’s /tmp/ocssw as a fallback).

ocsswroot = os.environ.setdefault("OCSSWROOT", "/tmp/ocssw")
env = Path(ocsswroot, "OCSSW_bash.env")
env.exists()

True

Our first OCSSW program is install_ocssw, which we use to print which version (tag) of OCSSW is installed. As just explained, we have to evaluate (or source) the environment configuration file first. To pass its location, we use {} variable expansion that is available with the ! prefix.

!source {env}; install_ocssw --installed_tag

WARNING: --tag is required to get the proper bundle list.

installedTag = V2025.2

The installedTag is our OCSSW version. Tags beginning with “V” are operational tags, while “T” tags are equivalent to a “beta” release and meant for testing by advanced users.

Setting up AWS S3 credentials#

Accessing data from NASA’s Earthdata Cloud, regardless of the tool, requires authentication. The earthaccess package works behind-the-scenes using the Earthdata Login credentials you provide to generate temporary AWS credentials for direct access to the Earthdata Cloud.

earthaccess.login(persist=True)
credentials = earthaccess.get_s3_credentials(provider="OB_CLOUD")

The OCSSW software accepts AWS credentials in all the usual methods, including via environment variables that we set in the next cell.

os.environ.update(
    {
        "AWS_ACCESS_KEY_ID": credentials["accessKeyId"],
        "AWS_SECRET_ACCESS_KEY": credentials["secretAccessKey"],
        "AWS_SESSION_TOKEN": credentials["sessionToken"],
    }
)

Earthdata Cloud sets a one-hour lifespan on your temporary AWS credentials. If you get an access denied error, re-run the cell above to generate new AWS credentials.

Navigating GIOP files in OCSSW#

In the ‘Run Level-2 Generator (l2gen) OCSSW program on OCI data’ tutorial, we learned how to navigate files within the OCSSW directory.

The IOP product suite#

Let’s take a look at the msl12_defaults_IOP.par file, which is the “parameter” file that defines the default IOP suite for OCI:

!cat $OCSSWROOT/share/oci/msl12_defaults_IOP.par

#
# IOP l2gen suite for OCI
#
proc_uncertainty=1
l2prod=a bb aph Kd adg_442 adg_s bbp_442 bbp_s rrsdiff aph_unc_442 adg_unc_442 bbp_unc_442
wavelength_3d=400,413,425,442,460,475,490,510,532,555,583,618,640,655,665,678,701

## IOP model for use in downstream products (e.g. Kd_###_lee)
##  7 = GIOP
iop_opt=7

Let’s discuss some of these parameters:

proc_uncertainty=1 : Rrs uncertainty propagation mode, where 1 = uncertainty propagation generating error variance (and 0 = no uncertainty propagation and 2 = uncertainty propagation generating full covariance matrix)
l2prod=a bb aph Kd adg_442 adg_s bbp_442 bbp_s rrsdiff aph_unc_442 adg_unc_442 bbp_unc_442 : a list of the geophysical products to be included in the L2 output file; not defining a wavelength outputs all defined wavelengths
wavelength_3d=400,413,425,442,460,475,490,510,532,555,583,618,640,655,665,678,701 : the wavelengths to be output
iop_opt=7 : the IOP model used for use in downstream product (e.g., spectral diffuse attenuation coefficients, where 7 = GIOP)

So, running the ‘IOP l2gen suite’ produces these L2 data products at 17 wavelengths unless otherwise defined.

GIOP attempts to simultaneously estimate the magnitudes of spectral backscattering by particles ($b_{bp}$(λ); m⁻¹), absorption by phytoplankton ($a_{ph}$(λ); m⁻¹), and the combined absorption by non-algal particles and colored dissolved organic material ($a_{dg}$(λ); m⁻¹). It assigns constant spectral values for seawater absorption and backscattering ($a_{w}$(λ) and $b_{bw}$(λ), respectively; m⁻¹), assumes spectral shape functions of the remaining constituent absorption and scattering components (those values denoted by an *), and retrieves the magnitudes of each remaining constituent (the Mₓ values in red) required to match the spectral distribution of $R_{rs}$(λ). The default configuration uses the following forward model:

\[ r_{rs}(\lambda) = G(\lambda) \cdot \frac{b_{bw}(\lambda) + {\color{red}M_{bp}}\, b^*_{bp}(\lambda)}{a_w(\lambda) + {\color{red}M_{ph}}\, a^*_{ph}(\lambda) + {\color{red}M_{dg}}\, a^*_{dg}(\lambda)} \]

$r_{rs}$(λ) is subsurface remote sensing reflectance and G(λ) is a catch-all term that accounts for bidirectional reflectance distributions and other sea and sky conditions.

GIOP’s default parameterization#

To find a couple other useful GIOP options, you can take a look at ‘common/msl12_defaults.par’ and ‘oci/msl12_defaults.par’.

As seen in these two files, the default GIOP parameterization is defined by:

giop_fit_opt=1 (GIOP model optimization method. 1 = Levenberg-Marquardt (L-M) fit)
giop_rrs_opt=0 (GIOP model Rrs to bb/(a+bb) method. 0 = Gordon quadratic; defines G(λ))
giop_aph_opt=2 (GIOP model aph function type. 2 = Bricaud 1998 aph spectrum; defines aph*(λ))
giop_adg_opt=1 (GIOP model adg function type. 1 = user-defined exponential adg function; defines adg*(λ))
giop_adg_s=0.018 (spectral parameter for adg fixed at 0.018; defines adg*(λ) with above)
giop_bbp_opt=3 (GIOP model bbp function type. 3 = power-law with exponent derived via Lee et al. (2002); defines bbp*(λ))
raman_opt=2 (Raman scattering Rrs correction options. 2 = Westberry et al. (2013) analytical correction)
giop_wave=[413,442,490,510,555,670] (wavelengths used in the L-M optimization)

Running l2gen on OCI data to produce the IOP data suite products will include these default options. The user must make modifications to produce different outputs.

Remember, you can find more information about these parameters, and the other options available for each, using ‘l2gen –help’.

Writing OCSSW parameter files#

User-generated parameter files provide a convenient way to control l2gen processing.

Without a par file, providing l2gen the names of the input L1B and output L2 files from the Terminal looks like this:

l2gen ifile=data/PACE_OCI.20250507T170659.L1B.V3.nc ofile=data/PACE_OCI.20250507T170659.L2.V3.nc

Alternatively, a user-defined par file, say “l2gen.par”, can be created with the following two lines of content:

ifile=data/PACE_OCI.20250507T170659.L1B.V3.nc
ofile=data/PACE_OCI.20250507T170659.L2.V3.nc

Now l2gen can now be called using the single argument par while generating the same result:

l2gen par=l2gen.par

You can imagine that the par file option becomes far more convenient when many changes from default are desired.

So let’s define a function to help write OCSSW parameter files, which is needed several times in this tutorial. To write the results in the format understood by OCSSW, this function uses the csv.writer from the Python Standard Library. Instead of writing comma-separated values, however, we specify a non-default delimiter to get equals-separated values. Not something you usually see in a data file, but it’s better than writing our own utility from scratch!

def write_par(path, par):
    """Prepare a parameter file to be read by one of the OCSSW tools.

    Using a parameter file (a.k.a. "par file") is equivalent to specifying
    parameters on the command line.

    Parameters
    ----------
    path
        where to write the parameter file
    par
        the parameter names and values included in the file
    """
    with open(path, "w") as file:
        writer = csv.writer(file, delimiter="=")
        writer.writerows(par.items())

back to top

2. Search and access L1B data#

Let’s use the earthaccess Python package to access a L1B file.

tspan = ("2025-05-07", "2025-05-07")
bbox = {"west": -76, "south": 35, "east": -74.5, "north": 39}

results = earthaccess.search_data(
    short_name="PACE_OCI_L1B_SCI",
    temporal=tspan,
    bounding_box=tuple(bbox.values()),
)
results[0]

Data: PACE_OCI.20250507T170659.L1B.V3.nc

Size: 1788.21 MB

Cloud Hosted: True

l1b_paths = earthaccess.open(results)
l1b_paths

[<File-like object S3FileSystem, ob-cumulus-prod-public/PACE_OCI.20250507T170659.L1B.V3.nc>]

And let’s do a quick plot of a rhot_red wavelength to see what the data looks like:

dataset = xr.open_datatree(l1b_paths[0])
dataset = xr.merge(dataset.to_dict().values())
dataset = dataset.set_coords(("longitude", "latitude"))
plot = dataset["rhot_red"].sel({"red_bands": 100}).plot()

../../_images/c50cf24ebfb62b6a4acc5325650df53c28e27dccfa5b6964ec9df582406c7343.png

back to top

Run the Generalized IOP (GIOP) inversion model in the Level-2 Generatorl2gen OCSSW program on OCI data

Contents

Run the Generalized IOP (GIOP) inversion model in the Level-2 Generator`l2gen` OCSSW program on OCI data#

Summary#

Learning Objectives#

Contents#

1. Setup#

Setting up AWS S3 credentials#

Navigating GIOP files in OCSSW#

The IOP product suite#

GIOP’s default parameterization#

Writing OCSSW parameter files#

2. Search and access L1B data#

3. Run `l2gen` with default GIOP configurations#

4. Plot L2 IOP data products#

5. Run GIOP with modifications to the configurations#

Run the Generalized IOP (GIOP) inversion model in the Level-2 Generatorl2gen OCSSW program on OCI data

Contents

Run the Generalized IOP (GIOP) inversion model in the Level-2 Generatorl2gen OCSSW program on OCI data#

Summary#

Learning Objectives#

Contents#

1. Setup#

Setting up AWS S3 credentials#

Navigating GIOP files in OCSSW#

The IOP product suite#

GIOP’s default parameterization#

Writing OCSSW parameter files#

2. Search and access L1B data#

3. Run l2gen with default GIOP configurations#

4. Plot L2 IOP data products#

5. Run GIOP with modifications to the configurations#

Run the Generalized IOP (GIOP) inversion model in the Level-2 Generator`l2gen` OCSSW program on OCI data#

3. Run `l2gen` with default GIOP configurations#