Understanding the PalmSens .pssession format

This post is part of Data Deep Dives, a new series where we’ll publish short articles about different instrument data file types we parse as part of integration with Labric.

Background

Although we work with labs across a very wide range of disciplines and research topics at Labric, some techniques are broadly useful enough that we see them all the time — electrochemical characterization with potentiostats certainly falls into this category. While discussing the range of possible techniques you might use a potentiostat or galvanostat to perform could be an article series on its own, you can enjoy this post with the minimal context that

1. potentiostats generally control potential and measure current, which can
2. be used to perform cyclic voltammetry (CV), electrochemical impedance spectroscopy (EIS), etc., typically applied to
3. problems ranging from battery performance quantification to ion-exchange mechanism investigation.

In this post, we focus on the PalmSens .pssession format, emitted by PalmSens’ PSTrace software. This format is used broadly across a number of their different potentiostat offerings, which mostly vary in terms of measurement specificity, size, and measurement range. The software stores all data related to a particular measurement in a .pssession file. Specific parts of the measurement (such as curves for a CV measurement) can be exported to Excel or as a csv using the software user interface.

It’s worth noting that PalmSens has a pretty great set of SDKs in multiple programming languages (Python, .NET, LabVIEW, MATLAB) which include functionality for both instrument control (such as starting an experiment) and data analysis (such as loading a .pssession file).

Motivation

“If the instrument manufacturer already provides exports in common formats and an SDK, why does this article even exist?” you might be asking.

Using the instrument manufacturer’s libraries is best as long as it fits your use case, but for our customers there have been a few reasons to go off the beaten path.

Exports from PSTrace are missing some key pieces of data

Although the CSV exports from PSTrace do contain measurement values and are sufficient for many use cases, they’re missing a lot of detail that .pssession files contain, such as detailed timestamps, measurement configuration details (such as range and duration), etc. — you might really want this data! Range and instrument metadata is also especially useful when you have multiple tools for the same technique and want to ensure measurements are comparable.

One other under-appreciated benefit of using .pssession files rather than exports is to avoid manual work. Even the simplest manual export step can be forgotten. Also, because a lot of metadata is missing in the PSTrace CSV exports, it’s common for researchers to compensate by putting information in the filename instead. In these cases, analysis can quickly become time-consuming as researchers have to correct for multiple people manually exporting and naming files differently.

The PalmSens SDKs have a .NET dependency

To use the PalmSens SDK you need to have .NET Framework 4.7.2 (for Windows) or .NET Runtime 9.0+ (for Mac/Linux) installed. The SDK wraps around the .NET libraries that are also used by PSTrace. Including .NET can add a lot of setup overhead, and is not needed to read the data.

When working with this data at Labric, our team set out to write a dependency-free, Python-only parser for .pssession files in an effort to keep processing fast and streamlined, the better to accommodate large data volumes. Because we’re only working with the files for data analysis, and not trying to connect to the instrument, this is very feasible.

The .pssession file format

Note: we’ve only looked at .pssession files from PSTrace version 5+, so use caution in applying these findings to older data.

At a glance

• Essentially JSON in terms of structure (with one caveat noted later in the Gotchas section)
• Contains a top-level “MethodForMeasurement” entry consisting of a “\r\n”-delimited list of KEY=VALUE pairs that have information about the measurement config, like potential range and any method-specific parameters (i.e. frequency range for EIS)
• The format puts unit details at the beginning of every series. All series in a measurement have the same number of data points and are accordingly aligned. The unit details block has information about the PalmSens type, the base symbol (”S”), the quantity name (”Q”), and the abbreviation to be used on the axis (”A”)

"Unit": {
       "Type": "PalmSens.Units.Volt",
       "S": "V",
       "Q": "Potential",
       "A": "E"
   },


• Each series contains a list with data points, each of which has multiple attributes:

 {
     "V": -59604.644775390625,
     "C": 7,
     "S": 2
 },
 
 # or possibly
 
 {
     "V": 0.0,
     "S": 0,
     "R": 7
 },


• “V” is present in all data point entries and has the value of the data point; for most purposes, this value with the unit info is sufficient to process the data contained in the file.
• For completeness, here’s what we’ve been able to determine about the other attributes. “S” is also present in all entries, and might indicate a change of range/resolution; it is usually zero. “C”/”R” are present in current and potential readings respectively, and “T” appears in GenericValue “hidden” series (series that don’t seem to be made available to the user in PSTrace) with a string payload that seems to contain debug info.
• Both raw measurement series and analysis series for plotting can be stored in one .pssession file, and the total number of series varies depending on the technique and analytical workflow used.

Some gotchas

• At first glance, the file contents look a lot like pure JSON. However, if you try to use json.loads() or other JSON-reading functions, you’ll find that it cannot be loaded directly; the file is UTF-16 and has a BOM at both ends. To load as JSON you’ll need to strip the trailing BOM:

   text = open(path, "rb").read().decode("utf-16")
   as_json = json.loads(text.strip("\ufeff"))


• The timestamps stored in .pssession files are .NET ticks, not POSIX timestamps, which the Python .fromtimestamp method expects. The .NET ticks count non-leap ten-millionths of a second elapsed since 00:00:00 UTC on January 1, 0001, while POSIX counts non-leap seconds elapsed since 00:00:00 UTC on Thursday, 1 January 1970.

DOTNET_TICKS_AT_UNIX_EPOCH = 621_355_968_000_000_000
DOTNET_TICKS_PER_SECOND = 10_000_000

seconds = (ticks - DOTNET_TICKS_AT_UNIX_EPOCH) / DOTNET_TICKS_PER_SECOND
as_datetime = datetime.fromtimestamp(seconds, timezone.utc)


• The .pssession file stores some exactly redundant information (both the raw dataset and data for any plotted curves), so be careful to avoid accidental duplication.

# as an example, in one file:
EISDataList[0]["DataSet"] == Measurements[0]["DataSet"]
MethodForMeasurement == Measurements[0]["Method"]


• The MethodForMeasurement string has KEY=VALUE pairs, but also includes “#”-prefixed comment lines to delineate different sections or hold more information.
• This is probably obvious to folks already working with PalmSens potentiostats, but different measurement types differ in terms of the kind of data that is saved. In particular, EIS measurements have some structural differences (EIS leaves the Curves list empty and puts data in EISDataList).
• Description fields (”scan1”) in series can be repeated across multiple quantities, so  quantities could easily be erased if code uses them to annotate and retain data, such as in a dict. ArrayType codes are kept consistent across quantities and are a great alternative for matching.
• The symbol in the Unit detail block (”S”) has the base unit symbol for the stored values, not the unit of the stored values! For example, a measurement with Unit[”S”] = “A” is not necessarily in amps, and might have Unit[”Type”] =”PalmSens.Units.MicroAmpere”. Be very careful when pulling in unit info.

# Something like this could work:
SI_PREFIX_SCALES = {
   "Femto": 1e-15, "Pico": 1e-12, "Nano": 1e-9, "Micro": 1e-6,
   "Milli": 1e-3, "Centi": 1e-2, "Kilo": 1e3, "Mega": 1e6, "Giga": 1e9,
}

unit_class = Unit["Type"]
match = re.search(r"PalmSens\.Units\.(%s)" % "|".join(SI_PREFIX_SCALES), unit_class)
scaled_to_si = SI_PREFIX_SCALES[match.group(1)] if match else 1.0


Example code

Though analysis and final storage of the data will vary from lab to lab and use case to use case, we’ve open-sourced a parser “skeleton” here to aid you in reading .pssession data with pure Python. Please reach out with any questions that come up when using this code as a starting point! Our goal is to support better data infrastructure for science across the board.

Closing thoughts

Through the course of our work with unusual or proprietary data formats, collaborators often ask whether it’s frustrating to have to process such a range of file types. Despite some of the aforementioned gotchas causing initial friction, the .pssession format was overall fairly nice to work with. Major positives include the inclusion of detailed units per series, the use of fairly descriptive section/series headers, and the existence of a publicly-available SDK, which was available as a backstop if any part of the file format was too opaque to read. The format being JSON-like is also an overall plus, although only a very minimal change would be needed to make it proper JSON (and more accessible). There were a few pain points when working with the format; the most confusing aspect initially was how much the output could vary between techniques, and the opaque per-data-point and per-series single letter attribute descriptors (”A”, “S”, “V”, etc.). One other point worth mentioning is data efficiency of this format. Because the JSON-like structure is used even for series’ data points, there’s about 80 bytes of .pssession data per single stored value, even excluding series duplication. This results in file sizes that are potentially many times bigger than a CSV with the majority of this data would require. For reference, we measured one pair of .pssession / .csv that describe the same measurement and found that the .pssession was about 4x the size of the .csv. Our team speculates that trying to keep the file size from ballooning further might have also been a motivating factor in choosing the aforementioned single-letter per-point descriptors.

Credit is also due to the team at PalmSens for their clear effort to support development over their tools and software. While instrument SDKs are becoming more common, it’s still a decent amount of extra work to develop and maintain them. Having these SDKs and their documentation available to all is already helpful and supports open science; the one area our team would have benefitted from more openness is in documentation of the PSTrace software functionality itself, which is only available behind a login portal and thus inaccessible to many.

We hope this writeup is helpful to others (or LLMs) working with .pssession data. If you’ve found it useful, or have further questions, we’d love to hear from you at blog@labric.co!

Q&A

When should you process this data directly rather than using the PalmSens SDK or exported CSVs?

As briefly mentioned above, I’d almost always recommend using the SDK for typical use cases; the one exception would be a situation like ours, where we’re only reading the data (not writing), and do not want to add a dependency on .NET if not needed. In our case, this choice kept startup time faster and image size smaller. Our team is also experienced with processing unusual file formats, and this instrument was in use across several customer labs, so the additional effort was feasible and worthwhile for us.

Why would PalmSens use .NET ticks for timestamps? It seems unnecessarily complicated and exclusive.

It likely isn’t intentionally exclusive. Most companies don’t design their instruments’ internal file format for consumption by tools outside of their software ecosystem, because most labs don’t use these formats unless they want to integrate into a more ambitious data infrastructure/ecosystem. For typical analysis workloads, the functionality offered by the instrument software is more than sufficient, and most people don’t need to look beyond that.

In the specific case you ask about, most likely it’s because their PSTrace software is written in .NET and this is how some internal writeTimestamp() function will output by default!

Why might PalmSens use a custom file type like .pssession in the first place?

The .pssession format is similar to other instrument formats in that it’s trying to keep information about the underlying measurement alongside both instrument metadata and context about how the user is analyzing the data. Usually, instrument manufacturers have both tabular-like data (measurement sweeps, in this case current vs. potential, etc.) as well as attribute data (properties of the instrument or measurement, in this case things like firmware version, maximum and minimum setpoints for sweep range, etc.). Keeping these different data “shapes” organized in a simple, commonly-used form is not easy.