---
title: "User Guide"
subtitle: "Package 'photobiologyInOut' `r packageVersion('photobiologyInOut')` "
author: "Pedro J. Aphalo"
date: "`r Sys.Date()`"
output: 
  rmarkdown::html_vignette:
    toc: yes
vignette: >
  %\VignetteIndexEntry{User Guide}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

## Introduction

Package 'photobiologyInOut' defines functions for importing spectral data from different
instruments, data repositories and simulation models, and for data exchange with R packages
'hyperSpec', 'colorSpec' and 'pavo' by reading or writing objects of classes 
defined in them (see Tables at the start of each section).

As of version 0.4.21 packages 'hyperSpec' and 'pavo' need to be installed only if the
corresponding functions will be used. This avoids bloat when
these functions are not needed. If the vignette is built or examples attempted to run in
a system where these packages are not installed, some output will be missing but
no error will be triggered.

All functions that read data from files attempt to decode and store as metadata the information present in file headers. In addition, in most cases the unchanged header of the file is stored unaltered as a comment in the constructed objects.

It should be remembered, though, that this package has been developed based on the _example files_ I had access to. Files from the same instruments with different hardware configurations, different firmware versions, or even settings may differ substantially. In many cases the output is produced by software in a host computer rather by the instrument itself, adding further uncertainties and possible differences due to for example the operating system of the host computer. A further complication is that in some cases the format of dates, times and numbers depends on the locale settings in use at the time of data acquisition, or analysis. For all those reasons, do expect to have to do some debugging, and most importantly always validate the imported data against the original file (remembering to run a new validation each time there is a software or firmware update) or update of this package as I test each version before release only with the example files I have available, which are not many.

## Preliminaries

```{r, setup, include=FALSE, cache=FALSE}
library(knitr)
# Are the packages used in examples installed?
eval_chunks <- requireNamespace("ggspectra", quietly = TRUE) &&
                 requireNamespace("photobiologyWavebands", quietly = TRUE)
# eval_colorSpec <- requireNamespace("colorSpec", quietly = TRUE) && eval_chunks
eval_colorSpec <- eval_chunks
eval_pavo <- requireNamespace("pavo", quietly = TRUE) && eval_chunks
eval_hyperSpec <- requireNamespace("hyperSpec", quietly = TRUE) && eval_chunks

opts_chunk$set(fig.align='center', fig.show='hold',
               fig.width=7, fig.height=6, size="footnotesize",
               eval=eval_chunks)

options(replace.assign = TRUE, width = 55,
        warnPartialMatchAttr = FALSE,
        warnPartialMatchDollar = FALSE,
        warnPartialMatchArgs = FALSE)
# setting TZ may be needed in some geographic locations as some Windows TZ 
# strings are not recognized by all versions of R
Sys.setenv(TZ = 'UTC')
```

When loading suggested packages flags are set based on success and later used to selectively evaluate code chunks.

```{r, message=FALSE}
library(photobiology)
library(photobiologyWavebands)
library(photobiologyInOut)
library(lubridate)
library(ggspectra)
library(readr)
library(colorSpec)
if (eval_pavo) {library(pavo)}
if (eval_hyperSpec) {library(hyperSpec)}
```

```{r}
# plot defaults
theme_set(theme_bw()) # ggplot2
set_annotations_default(annotations = c("+", "title:what:when"))
# decrease lines printed
options(tibble.print_max = 5,
        tibble.print_min = 3,
        photobiology.strict.range = NA_integer_)
```

## Spectrometer output files

In the examples we use function _system-file_ to reliably locate the example files included in the package. For reproducing the examples with these files, this call is needed. When using user's files only the path as a character string needs to be passed as argument.

We can also obtain the path to the folder were the example files are stored.

```{r}
extdata.path <- 
  system.file("extdata", 
              package = "photobiologyInOut", mustWork = TRUE)
```

In most cases the quantities in the files can be identified from the file contents allowing a single function to work adaptively. In cases when the format returned by different models of spectrometers from the same supplier is largely consistent, then a single function can read them all. In some cases the format depends on the computer software used to acquire the data, of which a single supplier may have had more than one over the years, thus requiring multiple functions. Some of the instruments supported have long been discontinued but in their time were very popular, like the LI-COR LI-1800 spectrometer from the late 1980's. The coverage of brands and models is limited by the instruments I have had access to, or have had at least access to a sample of files. As of version 0.4.24 data can be imported from files created by spectrometers and/or software from six different suppliers.

Functions for importing measured spectral data.

| R function             | Instrument             | Program           | `class` of value |
|:-----------------------|:-----------------------|:------------------|:-----------------|
| `read_oo_ssirrad()`    | Ocean Optics spectrom. | SpectraSuite      | `source_spct`    |
| `read_oo_ssdata()`     | Ocean Optics spectrom. | SpectraSuite      | `raw_spct`       |
| `read_oo_ovirrad()`    | Ocean Optics spectrom. | OceanView         | `source_spct`    |
| `read_oo_jazirrad()`   | Ocean Optics Jaz       | *instrument*      | `source_spct`    |
| `read_oo_jazpc()`      | Ocean Optics Jaz       | *instrument*      | `filter_spct`    |
| `read_oo_jazpc()`      | Ocean Optics Jaz       | *instrument*      | `reflector_spct` |
| `read_oo_jazdata()`    | Ocean Optics Jaz       | *instrument*      | `raw_spct`       |
| `read_oo_pidata()`     | Ocean Optics spectrom. | STS DK (Raspbian) | `raw_spct`       |
| `read_wasatch_csv()`   | Wasatch Phot. spectrom. | Enlighten        | `raw_spct`       |
| `read_wasatch_csv()`   | Wasatch Phot. spectrom. | Enlighten        | `filter_spct`    |
| `read_wasatch_csv()`   | Wasatch Phot. spectrom. | Enlighten        | `reflector_spct` |
| `read_wasatch_csv()`   | Wasatch Phot. spectrom. | Enlighten        | `source_spct`    |
| `read_cid_spectravue_csv()` | CID Bio-Science CI-710s | *instrument* | `filter_spct`   |
| `read_cid_spectravue_csv()` | CID Bio-Science CI-710s | *instrument* | `reflector_spct` |
| `read_cid_spectravue_csv()` | CID Bio-Science CI-710s | *instrument* | `object_spct`   |
| `read_avaspec_csv()`   | Avantes spectrom.      | *instrument?*     | `source_spct`    |
| `read_asd_txt()`   | ASD spectrom.      | *software: .asd -> text* | `source_spct`     |
| `read_asd_txt()`   | ASD spectrom.      | *software: .asd -> text* | `reflector_spct`  |
| `read_asd_txt()`   | ASD spectrom.      | *software: .asd -> text* | `filter_spct`     |
| `read_asd_txt()`   | ASD spectrom.      | *software: .asd -> text* | `generic_spct`    |
| `read_macam_file()`    | Macam                  | *instrument*      | `source_spct`     |
| `read_li180_txt()`    | LI-COR LI-180          | *instrument*      | `source_spct`     |
| `read_m_li180_txt()`  | LI-COR LI-180          | *instrument*      | `source_mspct`    |
| `read_licor_prn()`    | LI-COR LI-1800         | PC1800 (MS-DOS)   | `source_spct`     |
| `read_licor_prn()`    | LI-COR LI-1800         | PC1800 (MS-DOS)   | `filter_spct`     |
| `read_licor_prn()`    | LI-COR LI-1800         | PC1800 (MS-DOS)   | `reflector_spct`  |
| `read_m_licor_prn()` | LI-COR LI-1800         | PC1800 (MS-DOS)   | `source_mspct`    |
| `read_m_licor_prn()` | LI-COR LI-1800         | PC1800 (MS-DOS)   | `filter_mspct`    |
| `read_m_licor_prn()` | LI-COR LI-1800         | PC1800 (MS-DOS)   | `reflector_mspct` |
| `read_spectrapen_csv()` | PSI SpectraPen      | *instrument* (?)    | `source_spct`   |

### Ocean Optics spectrometers

Most Ocean Optics array spectrometers are used tethered to a computer, either locally through USB, or remotely through Ethernet. The Jaz modular spectrometer was an exception to this as it can be used autonomously through a simple built-in keyboard and a minuscule display. In addition to these two programs, software for use on Raspberry Pi SBCs was available for some time. See packages 'rOmniDriver' and 'ooacquire' for support of data acquisition directly from within R.

Example data files from Ocean Optics' spectrometers and software have names starting with `oo-`:

```{r}
list.files(extdata.path, pattern = "^oo-")
```


#### SpectraSuite software

The _SpectraSuite_ program supported nearly all Ocean Optics array spectrometers returned similar text files for all of them. However, this program was later replaced by the _OceanView_ program.

A file from an Ocean Optics' Q6500 spectrometer, with data processed with the SpectraSuite software.

Format of the header is similar, but not identical. The first few lines of the file look like this:

```
SpectraSuite Data File
++++++++++++++++++++++++++++++++++++
Date: Mon May 06 15:13:40 CEST 2013
User: User
Dark Spectrum Present: Yes
Reference Spectrum Present: No
Number of Sampled Component Spectra: 1
Spectrometers: QEB1523
Integration Time (usec): 100000 (QEB1523)
Spectra Averaged: 1 (QEB1523)
Boxcar Smoothing: 0 (QEB1523)
Correct for Electrical Dark: No (QEB1523)
Strobe/Lamp Enabled: No (QEB1523)
Correct for Detector Non-linearity: No (QEB1523)
Correct for Stray Light: Yes (QEB1523)
Number of Pixels in Processed Spectrum: 1044
>>>>>Begin Processed Spectral Data<<<<<
199.08	0.0000E00
199.89	0.0000E00
200.70	0.0000E00
...
```

```{r}
q.raw.file <- 
  system.file("extdata", "oo-spectrum.SSIrrad", 
              package = "photobiologyInOut", mustWork = TRUE)
autoplot(read_oo_ssirrad(file = q.raw.file))
```

#### OceanView software

SpectraSuite was replaced by the currently available OceanView program. Although the output files saved with this program are similar to those saved by SåectrSuite, they differ enough to require different decoding of metadata.

```
Data from zoltapolka77percent_AbsoluteIrradiance__0__19-20-35-388.txt Node

Date: Thu Jun 19 19:20:35 CEST 2025
User: DM
Spectrometer: HR600768
Trigger mode: 0
Integration Time (sec): 1,000000E-1
Scans to average: 5
Nonlinearity correction enabled: true
Boxcar width: 0
Storing dark spectrum: false
XAxis mode: Wavelengths
Number of Pixels in Spectrum: 2048
>>>>>Begin Spectral Data<<<<<
178,699	0
179,053	0
179,407	0
```

```{r}
q.raw.file <- 
  system.file("extdata", "oo-spectrum.OVIrrad", 
              package = "photobiologyInOut", mustWork = TRUE)
autoplot(read_oo_ovirrad(file = q.raw.file, 
                         locale = readr::locale("en", 
                                                decimal_mark = ",",
                                                grouping_mark = "",
                                                tz = "Europe/Warsaw")),
         range = c(300, NA))
```

#### Jaz: Raw detector counts

Contrary to other OceanOptics array spectrometers, the Jaz modular spectromter is autonomous and can save spectral data directly to memory cards. The format of these files is similar but not the same as those described above.

Reading a raw data file generated by Ocean Optics' Jaz spectrometer is the first example. The light source was the Jaz PX pulsed Xenon light module. 

The first few lines of the file look like this, with W for wavelength, D for dark, R for reference, S for sample and P for processed (all spectral data values are raw detector counts):

```
Jaz Data File
++++++++++++++++++++++++++++++++++++
Date: Mon Apr 25 12:49:11 2016
User: jaz
Dark Spectrum Present: Yes
Reference Spectrum Present: Yes
Processed Spectrum Present: Yes
Spectrometers: JAZA3098
Integration Time (usec): 748000 (JAZA3098)
Spectra Averaged: 1 (JAZA3098)
Boxcar Smoothing: 0 (JAZA3098)
Correct for Electrical Dark: No (JAZA3098)
Strobe/Lamp Enabled: Yes (JAZA3098)
Correct for Detector Non-linearity: No (JAZA3098)
Correct for Stray Light: No (JAZA3098)
Number of Pixels in Processed Spectrum: 2048
>>>>>Begin Processed Spectral Data<<<<<
W	D	R	S	P
190.313904	0.000000	0.000000	0.000000	0.000000
190.695511	0.000000	0.000000	0.000000	0.000000
191.077087	1138.953125	1123.134277	1102.795898	228.570541
191.458633	1184.149658	1227.086426	1059.859131	-289.473419
191.840149	1175.110352	1193.188965	1132.173584	-237.500336
...

```

```{r}
jaz.raw.file <- 
  system.file("extdata", "oo-spectrum.jaz", 
              package = "photobiologyInOut", mustWork = TRUE)
jazraw.spct <- read_oo_jazdata(file = jaz.raw.file)
jazraw.spct <- trim_wl(jazraw.spct, range = c(250, 900))
```

Plotting the spectrum.

```{r}
autoplot(jazraw.spct)
```

The metadata stored in attributes can be accessed with functions. It is clear, that not all settings can be recovered from the file. However, we store the record will all the fields which would have been filled if the data had been acquired directly from R using package 'ooacquire'.

```{r}
when_measured(jazraw.spct)
```

```{r}
getInstrDesc(jazraw.spct)
```

```{r}
getInstrSettings(jazraw.spct)
```

#### Jaz: Spectral energy irradiance

Reading an "Absolute Irradiance File" (sic) generated by Ocean Optics' Jaz spectrometer results in a `source_spct` object. In this example, the light source measured was a `white' fluorescent tube.

The first few lines of the file look like this:
```
Jaz Absolute Irradiance File
++++++++++++++++++++++++++++++++++++
Date: Tue Feb 03 09:44:41 2015
User: jaz
Dark Spectrum Present: Yes
Processed Spectrum Present: Yes
Spectrometers: JAZA1065
Integration Time (usec): 193000 (JAZA1065)
Spectra Averaged: 3 (JAZA1065)
Boxcar Smoothing: 5 (JAZA1065)
Correct for Electrical Dark: Yes (JAZA1065)
Strobe/Lamp Enabled: No (JAZA1065)
Correct for Detector Non-linearity: Yes (JAZA1065)
Correct for Stray Light: No (JAZA1065)
Number of Pixels in Processed Spectrum: 2048
Fiber (micron): 3900
Collection Area: 0.119459
Int. Sphere: No
>>>>>Begin Processed Spectral Data<<<<<
W	D	S	P
188.825226	0.000000	0.000000	0.000000
189.284851	0.000000	0.000000	0.000000
189.744415	-89.659378	-90.917900	-0.000000
190.203964	-106.165916	-96.419785	0.000000
...
```

```{r}
jaz.s.irrad.file <- 
  system.file("extdata", "oo-spectrum.jazIrrad", 
              package = "photobiologyInOut", mustWork = TRUE)
jaz.spct <- read_oo_jazirrad(file = jaz.s.irrad.file)
jaz0.spct <- jaz.spct
jaz.spct <- trim_wl(jaz.spct, range = c(290, 800))
```

Plotting the spectrum.

```{r}
autoplot(jaz.spct)
```

#### "Cleaning" spectral data

We can see that the data have problems. We get a warning because the data include
negative values for spectral irradiance. We will use some methods from package
'photobiology' to correct the problem. As the data are noisy we cannot just shift
the scale so that the most negative value becomes zero. Neither can we replace
all negative values with zeros, as this would create bias.

In the following code chunk we will use a region of the spectrum in which
spectral irradiance is known to be equal to zero as reference to  shift the
scale zero. Afterwards we discard data ``known'' to be zero, and for which the instrument calibration is not valid, and finally we plot the spectrum.

```{r}
jaz.spct <- fshift(jaz0.spct, range = c(255, 290), f = "mean")
jaz.spct <- trim_wl(jaz.spct, range = c(290, 800))
autoplot(jaz.spct)
```

We can next try to smooth the spectrum as it is very noisy outside the visible region.

```{r}
jaz.spct <- smooth_spct(jaz.spct)
autoplot(jaz.spct)
```

Photon and energy irradiances.

```{r}
e_irrad(jaz.spct, PAR())       # W m-2
```


All in one statement.

```{r}
autoplot(read_oo_jazirrad(file = jaz.s.irrad.file))
```

As above but limiting the wavelength range plotted.

```{r}
autoplot(read_oo_jazirrad(file = jaz.s.irrad.file),
     range = c(250,850))
```

Adding our custom ``adaptive'' smoothing.

```{r}
autoplot(smooth_spct(read_oo_jazirrad(file = jaz.s.irrad.file)),
     range = c(250,850))
```

### Wasatch Photonics array spectrometers

The _Enlighten_ program from Wasatch Photonics can save spectral data in a variety of file formats. The function `read_wasatch_csv()` reads CSV files with spectral data in columns. It is designed so that it can read any variation of this file format. In _Enlighten_ it is possible to select which columns are included in the file so their number can vary. However, the header is rich in information and this allows in many but not all cases to guess based on the `"Technique"` used in _Enlighten_ the type of data being imported. This means that in some cases the user needs to pass an argument to parameter `s.qty`. This is also the case when the column to be extracted is not that with heading "Processed" in the CSV file. The metadata is as for Ocean Insight/Ocean Optics spectrometers copied to attributes in the returned object.

Example files:
```{r}
list.files(extdata.path, pattern = "^wasatch-")
```

Format of the header is simple and rather easy to parse. The header of the file and the first few lines look like this:
```
ENLIGHTEN Version,2.2.7
Measurement ID,20211003-134004-612972-WP-00591
Serial Number,WP-00591
Model,WP-UV-VIS-C-S-25
Label,13:40:04 WP-00591
Declared Match,
Declared Score,0
Scan Averaging,1
Boxcar,0
Technique,Scope
Baseline Correction Algo,
ROI Pixel Start,0
ROI Pixel End,1023
Slit Width,25
Vignetted,False
Interpolated,False
Raman Intensity Corrected,False
Deconvolved,False
Integration Time,60
Timestamp,2021-10-03 13:40:04.612972
Note,Dark substracted
Temperature,-15.327480412352088
CCD C0,247.9385986328125
CCD C1,0.5131465792655945
CCD C2,-0.00012270470324438065
CCD C3,7.728250039917839e-08
CCD Offset,0
CCD Gain,1.9
Laser Wavelength,0.0
Laser Enable,False
Laser Power,100
Laser Temperature,0
Pixel Count,1024

Pixel,Wavelength,Processed
0,247.94,175.00
1,248.45,178.00
```

```{r}
file.name <- 
    system.file("extdata", "wasatch-enlighten-scope.csv",
                package = "photobiologyInOut", mustWork = TRUE)
              
wasatch.raw.spct <- 
    read_wasatch_csv(file = file.name, extra.cols = "drop")
```

```{r}
summary(wasatch.raw.spct)
```

```{r}
autoplot(wasatch.raw.spct)
```

### Avantes array spectrometers

Example files:
```{r}
list.files(extdata.path, pattern = "^avantes-")
```

Avantes' two column .csv files can also be imported.

```{r}
ava.raw.file <- 
  system.file("extdata", "avantes-avaspec-e-irrad.csv", 
              package = "photobiologyInOut", mustWork = TRUE)
autoplot(read_avaspec_csv(file = ava.raw.file),
     range = c(280, 900), unit.out = "photon")
```

### ASD array spectrometers 

Data from ASD spectrometers is saved into binary files with name tag `.asd`. Software is available for conversion into tab-separated files of comma separated files. Function `read_asd_txt()` supports the import of tab-separated files. It extracts metadata from the file header and uses it to guess the quantity stored and then constructs an objects of one the classes from package 'photobiology'.

Example files:
```{r}
list.files(extdata.path, pattern = "^asd-")
```

The whole header and the first few rows of data from one of these files is shown below. The data are already expressed as expected by the functions from package 'photobiology'.

```

Text conversion of header file \h820\optphotolab\Drones\DJI__Mavic_3M\viikki-field_calibration__20240521\sky_irrad\irrad_sky__00073.asd
 -------------------------------------------------------------
                                                                                                                                                            
The instrument number was  18875/1
New ASD spectrum file: Program version = 6.04 file version = 7.00
Spectrum saved: 05/21/2024 at 11:33:07
VNIR integration time : 136
VNIR channel 1 wavelength = 350 wavelength step = 1
There was one sample per data value
xmin = 350 xmax= 2500
ymin= 0 ymax= 2.5
The instrument digitizes spectral values to 16 bits
SWIR1 gain was 16 offset was 2051
SWIR2 gain was 16 offset was 2070
Join between VNIR and SWIR1 was 1000 nm
Join between SWIR1 and SWIR2 was 1800 nm
VNIR dark signal subtracted
100 dark measurements taken Tue May 21 11:27:15 2024
DCC value was 0
Data is not compared to a white reference
There was a remote cosine receptor attached
Spectrum file is raw data with embedded reference
GPS-Latitude is S0
GPS-Longitude is E0
GPS-Altitude is 0
GPS-UTC is 00:00:00

Smart Detector
Serial#  0
Signal (A)  0.000E00
Dark (A)  0.000E00
Ref (A)  0.000E00
Status  0 - Uninitialized
Gain  E-4
Averaging  0
Temp (C)  0.0
Humid (%)  0.0

Wavelength	rad_sky-00073.asd
350	 0.219269418855826 
351	 0.215215995270273 
352	 0.213627983394367 
353	 0.220254134692977 
```
Energy irradiance, from the file shown above.

```{r}
asd.irrad.file <- 
  system.file("extdata", "asd-e-irrad-sky.tsv", 
              package = "photobiologyInOut", mustWork = TRUE)
read_asd_txt(file = asd.irrad.file,
             tz = "Europe/Helsinki") |> 
  clip_wl(c(NA, 2200)) |> 
  autoplot()
```

Reflectance, from a different file. It is, however, not possible to extract from the header informatio to distinguish total, specular or at other angles of acceptance or direction measurements.

```{r}
asd.reflectance.file <- 
  system.file("extdata", "asd-Rfr-soil.tsv", 
              package = "photobiologyInOut", mustWork = TRUE)
read_asd_txt(file = asd.reflectance.file,
             tz = "Europe/Helsinki") |>
  clip_wl(c(NA, 1350)) |>
  autoplot()
```

### MACAM scanning spectrometers

Macam's (now Iridian) single column DTA files can also be imported.

Example files:
```{r}
list.files(extdata.path, pattern = "^macam-")
```

The first few lines of the file look like this with all data in a single column with alternate rows for wavelengths (in nm) and irradiances, and a very terse header:

```
@19/5/1997
@17:44:58
#No Title
 2.5000000000E+02
 0.0000000000E+00
 2.5100000000E+02
 0.0000000000E+00
 2.5200000000E+02
 0.0000000000E+00
...
```

```{r}
macam.raw.file <- 
  system.file("extdata", "macam-e-irrad.DTA", 
              package = "photobiologyInOut", mustWork = TRUE)
autoplot(read_macam_dta(file = macam.raw.file))
```

### LI-COR spectrometers

Example files:
```{r}
list.files(extdata.path, pattern = "^licor-")
```

#### LI-180 array dector spectrometer from LI-COR

The LI-COR LI-180 is a portable, self-contained instrument, with relatively low
spectral resolution. This instrument measures spectral irradiance in the range 
380 nm to 780 nm. There is only one hardware configuration but firmware updates
have been released.

This instrument can save the spectral data in different formats. The "XXX" 
format includes a rather long header, followed by spectral data, then followed 
by a file footer. The long header can contain various summaries computed from
the spectrum, as well as date and time, and information on the instrument.

The top of the file we will read looks like this:

```
Model Name	LI-180
Serial Number	A18M0157
Time	2021/03/02_09:24:26
PPFD	129.879440
PFD	171.178452
PFD-UV	2.336031
PFD-B	33.417435
PFD-G	46.992531
PFD-R	49.469433
PFD-FR	38.963074
Custom1(655~665nm)	5.057602
Custom2(725~735nm)	4.957998
Custom3(650~670nm)	10.137251
Custom4(720~740nm)	9.965228
UV%	1.364676
B%	19.521980
G%	27.452354
R%	28.899324
FR%	22.761662
Custom1%	2.954579
Custom2%	2.896391
Custom3%	5.922037
Custom4%	5.821544
R:B	1.480348
R:FR	1.269649
R:G	1.052708
B:G	0.711122
UV:B	0.069905
UV:FR	0.059955
B:G:R	0.000000
B:R:FR	0.000000
UV:B:G:R:FR	0.000000
Ratio1(Custom1:Custom2)	1.020090
Ratio2(Custom3:Custom4)	1.017262
Ratio3	0.000000
Ratio4	0.000000
LambdaP	495.000000
LambdaPV	106.211479
LambdaD	554.000000
LUX	7366.741699
IRR	35.163498
fc	684.641418
I-Time	92.000000
380nm	27.503876
381nm	27.968113
382nm	28.452974
383nm	28.665405
...
```
And the footer containing chromaticity data:

```
...
778nm	82.020149
779nm	82.078781
780nm	82.042412
CCT	5465.000000
Duv	0.004667
x	0.333318
y	0.351069
u'	0.203671
v'	0.482666
deltax	0.000063
deltay	0.009303
deltau'	-0.003491
deltav'	0.004648
Purity	5.337072
CRI	98.779282
R1	98.665459
R2	99.470703
R3	98.974945
R4	98.248100
R5	98.830368
R6	99.473335
R7	98.902351
R8	97.668961
R9	94.101921
R10	99.232346
R11	98.246521
R12	99.336227
R13	98.899147
R14	99.324699
R15	97.831963
```

Function `read_licor_espd()` will automatically extract the spectral data, 
date and time, and serial number.

```{r}
licor_espd.file <- 
  system.file("extdata", "licor-li180-irrad.txt", 
              package = "photobiologyInOut", mustWork = TRUE)
li180.spct <- read_li180_txt(file = licor_espd.file)
```

In all cases as much information as possible is decoded, and the data file headers
are preserved as comments in the source.spct objects.

```{r}
li180.spct
cat(comment(li180.spct))
getInstrDesc(li180.spct)
getInstrSettings(li180.spct)
autoplot(li180.spct, unit.out = "photon")
```

#### LI-1800 scanning spectrometer from LI-COR

The LI-COR LI-1800 was developed in the early 1980's and remained available for
many years. Some units are still in use although the technology has been
superceeded. If re-calibrated these instruments are still useful.

Spectral photon irradiance output files generated by LI-COR's PC1800 program for the LI-1800 spectroradiometer. These files have a relatively detailed header, but it lacks year information. Files can contain either energy or photon based spectral irradiances, and this is signalled in the header. In this example photon (= quantum) spectral irradiance is returned. The first few lines of the file look like this:

```
"FILE:FL2"
"REM: TLD 36W/865       (QNTM)"
"LIMS: 300- 900NM"
"INT:  1NM"
"DATE:08/23 16:32"
"MIN:  300NM  1.518E-04"
"MAX:  546NM  7.491E-01"
 300  1.518E-04
 301  3.355E-04
 302  2.197E-04
 303  3.240E-04
...
```

Function `read_licor_prn` will automatically detect whether the data is energy or photon based. In all cases as much metadata information as possible is decoded, and the data file headers are preserved as comments in the source.spct objects. **The missing information for year is set to zero in the `when.measured` attribute, with month, day, hours and minutes as decoded from the header. The time zone defaults to UTC and will need in general to be passed as an argument to `tz` in the function call.**

```{r}
licor.file <- 
  system.file("extdata", "licor-li1800-q-irrad.PRN", 
              package = "photobiologyInOut", mustWork = TRUE)
licor.spct <- read_licor_prn(file = licor.file, tz = "Europe/Helsinki")
```

Even when using to correct argument for `tz` the time will still default to UTC when the spectrum is printed or plotted, but this time expressed in UTC may still be shifted from the correct time in the time zone where measurements were acquired as without year information it is impossible to adjust for daylight saving time and other shifts in local times that have changed over the years as a result of changes in legislation. 

```{r}
licor.spct
cat(comment(licor.spct))
autoplot(licor.spct, unit.out = "photon")
```

It is also possible to use the same function to import reflectance, and 
transmittance spectra acquired by the LI-1800.

And a spectral reflectance output file generated by LI-COR's PC1800 program for the LI-1800 spectroradiometer is used next.

The first few lines of the file look like this:

```
"FILE:RGD1"
"REM: REFL GREEN AD 1 "
"LIMS: 350- 800NM"
"INT:  2NM"
"DATE:05/30 13:50"
"MIN:  358NM  4.628E-02"
"MAX:  776NM  4.693E-01"
 350  5.135E-02
 352  4.713E-02
 354  5.324E-02
 356  4.740E-02
...
```

Function `read_licor_prn` cannot automatically detect the spectral quantity in the file, and when the irradiance default is not correct, users
need to override it with an explicit argument for parameter `s.qty`.

```{r}
licor.file <- 
  system.file("extdata", "licor-li1800-Rfr.PRN", 
              package = "photobiologyInOut", mustWork = TRUE)
licor.spct <- read_licor_prn(file = licor.file, s.qty = "Rfr")
```

In all cases as much information as possible is decoded, and the data file headers
are preserved as comments in the source.spct objects.

```{r}
licor.spct
cat(comment(licor.spct))
autoplot(licor.spct)
```

### PSI SpectraPen spectroradiometer

This all-in-one spectroradiometer measures spectral irradiance, and returns the data in a `.CSV` file with one column per spectrum. The data are returned expressed in energy or photon based units.

Example files:
```{r}
list.files(extdata.path, pattern = "^psi-")
```

The first few lines of a data block look like this:

```
Irradiance 	 [µW/cm2/nm],
Time,11/10/2022 7:08:01 PM,11/10/2022 7:08:05 PM,11/10/2022 7:08:10 PM,11/10/2022 7:08:32 PM,11/10/2022 7:08:36 PM,11/10/2022 7:08:42 PM,
Index,13,14,15,16,17,18,
Name,,,,,,,
GPS,,,,,,,
[nm],
327.1,3.733005E+000,3.711488E+000,3.914620E+000,9.741472E+000,9.176950E+000,1.050528E+001
329,3.557857E+000,3.256521E+000,3.686638E+000,8.320147E+000,8.764947E+000,8.995773E+000
330.8,3.250507E+000,2.988691E+000,3.100541E+000,7.933364E+000,8.187113E+000,8.819886E+000
...
```
or

```
Irradiance 	 [µE/m2/s/nm],

Time,11/10/2022 7:08:01 PM,11/10/2022 7:08:05 PM,11/10/2022 7:08:10 PM,11/10/2022 7:08:32 PM,11/10/2022 7:08:36 PM,11/10/2022 7:08:42 PM,
Index,13,14,15,16,17,18,
Name,,,,,,,
GPS,,,,,,,
[nm],
327.1,1.020666E-001,1.014783E-001,1.070323E-001,2.663483E-001,2.509133E-001,2.872321E-001
329,9.783817E-002,8.955168E-002,1.013796E-001,2.287973E-001,2.410289E-001,2.473764E-001
330.8,8.989830E-002,8.265734E-002,8.575075E-002,2.194107E-001,2.264286E-001,2.439290E-001
```

```{r}
  file.name <- 
    system.file("extdata", "psi-spectrapen-SP.csv", 
                package = "photobiologyInOut", mustWork = TRUE)
  psi.mspct <- read_spectrapen_csv(file = file.name,
                                  tz = "UTC")
  summary(psi.mspct)
  autoplot(psi.mspct, annotations = "")
```
```{r}
summary(psi.mspct[["spct.14"]])
autoplot(psi.mspct[["spct.14"]])
```

**Note:** It is clear from the figure above that this spectrometer suffers badly from straylight in the UV-A region, and readings at wavelengths shorter than 400 nm in the presence of stronger radiation at longer wavelengths are to be discarded.

### CID Bio-Science leaf spectrometer

This all-in-one spectrometer measures spectral reflectance, spectral transmittance or spectral absorbance of plant leaves or in fact any thin film. There is a single configuration available and being based on a microcontroller this instrument is used autonomously and spectral data saved internally can be exported as CSV files.

Reflectance and transmittance generate a single spectrum per measurement while absorbance generates three: spectral absorbance, spectral reflectance and spectral transmittance. The last two are needed to compute absorbance, and, wisely, they are also returned. In the case of absorbance measurements it is possible to import the data into an `object_spct` containing variables `Rfr` and `Tfr` or as separate objects.

Example files:
```{r}
list.files(extdata.path, pattern = "^cid-")
```

```{r}
  file.name <- 
    system.file("extdata", "cid-spectravue-Rpc-Measurements.csv", 
                package = "photobiologyInOut", mustWork = TRUE)
  cid_Rpc.spct <- read_cid_spectravue_csv(file = file.name)
  summary(cid_Rpc.spct)
  autoplot(smooth_spct(cid_Rpc.spct, method = "supsmu"), 
           range = c(400, 1000), annotations = "") %+%
    ylim(0, 0.55)
```


## Data from loggers

### Campbell Scientific

[Campbell Scientific](https://www.campbellsci.eu/) is a well know supplier of data loggers for commercial and
research applications.
Function `read_csi_dat()` defined in this package has been tested with a recent datalogger
model, the CR6, and using recent versions of programs PC400 and PC200W to
download the data. The currently used format of .DAT files is easy to decode
and our function can automatically detect the number and type of columns and
the number of rows.

Example files:
```{r}
list.files(extdata.path, pattern = "^campbellsci-")
```

```{r}
cs.day.file <- 
  system.file("extdata", "campbellsci-cr6-day.dat", 
              package = "photobiologyInOut", mustWork = TRUE)
```

Executing the statement below displays the 10 top lines of the DAT file as is, one character string per line.

```{r, eval=FALSE}
# not run
read_lines(cs.day.file, n_max = 10)
```

```{r}
day.dat <- read_csi_dat(file = cs.day.file)
day.dat
```

All information is preserved in the returned
`tibble::tibble` object, which is derived from `data.frame`.

```{r}
cs_hour.file <- 
  system.file("extdata", "campbellsci-cr6-hour.dat", 
              package = "photobiologyInOut", mustWork = TRUE)
hour.dat <- read_csi_dat(file = cs_hour.file)
ggplot(hour.dat, aes(TIMESTAMP, PAR_Den_Avg)) + geom_line()
```

### YoctoPuce

[Yocto Puce Sarl](https://www.yoctopuce.com/) sells numerous different USB modules. Those capable of data
input can log the data to memory and these data can be downloaded as a CSV
file. These files can be easily read into R using base functions but function `read_yoctopuce_csv()` makes this even a bit simpler.

Example files:
```{r}
list.files(extdata.path, pattern = "^yoctopuce-")
```

```{r}
yoctopuce_hour.file <- 
  system.file("extdata", "yoctopuce-data.csv", 
              package = "photobiologyInOut", mustWork = TRUE)
```

Executing the statement below displays the 10 top lines of the CSV file as is, one character string per line.

```{r, eval=FALSE}
# not run
read_lines(yoctopuce_hour.file, n_max = 10)
```

Here we import and plot the data.

```{r}
hour.dat <- read_yoctopuce_csv(file = yoctopuce_hour.file)
ggplot(hour.dat, aes(ISO.time, temperature.avg)) + geom_line()
```

## Output from simulation models

Functions for importing simulated spectral data.

| R function             | Simulation model       | Version           | `class` of value |
|:-----------------------|:-----------------------|:------------------|:-----------------|
| read_tuv_usrout()    | TUV (S. Madronich)     | version 5.0       | source_spct     |
| read_tuv_usrout2mspct()    | TUV (S. Madronich)     | version 5.0       | source_mspct     |
| read_qtuv_txt()    | TUV (S. Madronich)     | version 5.2       | source_spct     |
| read_uvspec_disort() | libRadtran             | irradiance        | source_spct     |
| read_uvspec_vesa()   | (T. & V. Kotilainen)   | irradiance        | source_spct     |
| read_fmi_cum()       | (A. Lindfors)          | daily cumulated   | source_spct     |
| read_m_fmi_cum()    | (A. Lindfors)          | daily cumulated   | source_mspct    |

### TUV

The output from the TUV model can be imported either by editing it before
import, or by making a simple edit to the output routine of TUV. This function
is known to work with TUV version 5.0 output. The output from TUV can contain
a variable number of spectra in ''parallel'' columns, which are _melted_
into a single column, with a factor with letters as levels, a numeric variable
with the zenith angle and a POSIXct column with times. A date needs to be
always supplied as the output file from TUV has only time of day information.

Example files:
```{r}
list.files(extdata.path, pattern = "^tuv-")
```

```{r}
tuv.file <- 
  system.file("extdata", "tuv-usrout.txt", 
              package = "photobiologyInOut", mustWork = TRUE)
tuv.spct <- read_tuv_usrout(file = tuv.file,
                            date = ymd("2014-03-21"))
summary(subset(tuv.spct, spct.idx == "A"))
tuv.spct
```

It is possible to extract individual spectra with subset, or as done here
plot them in different panels.

```{r, fig.height=10}
autoplot(tuv.spct, annotations = c("colour.guide")) +
  facet_wrap(~as.character(date), ncol = 2)
```

The output is a single `source_spct} object that can be easily converted
into a `source_mspct} object containing the individual spectra as
members of the collection.

```{r}
tuv.mspct <- subset2mspct(tuv.spct)
summary(tuv.mspct)
autoplot(tuv.mspct)
```

A file can be directly read into a collection using `read_tuv_usrout2mspct()`
which is a simple wrapper.

With the default of `lubridate::today()` date times are 'mapped' to
the current local date using the time zone of the computer as visible to R.
**This is unlikely to be correct!**

```{r}
tuv_nd.spct <- read_tuv_usrout(file = tuv.file)
when_measured(tuv_nd.spct)
```

### Quick TUV calculator

The files output by the online calculator based on the TUV model, contain at
most one spectrum, and arguments to only some parameters can be set by users.
However, it is convenient to use when we only need a few simple simulations.

Example files:
```{r}
list.files(extdata.path, pattern = "^qtuv-")
```

Function `read_qtuv_txt()` can extract spectra and the corresponding metadata
from these files. These files do contain date time information and geolocation
data when they are supplied as arguments to the calculator interface, otherwise
only zenith angle is available.

```{r}
qtuv.file <- 
  system.file("extdata", "qtuv.txt", 
              package = "photobiologyInOut", mustWork = TRUE)
qtuv.spct <- read_qtuv_txt(file = qtuv.file)
summary(qtuv.spct)
qtuv.spct
```

### libRadtran

By default libRadtran's uvspec writes only spectral irradiances to a text file as output. This is different from 'TUV' which by default includes an extensive header with the parameter settings used for the simulation. It is easy to read this simple output file with R's
functions. However, we provide functions, that simplify reading of the files. The
output from uvspec varies depending on its input. The main source of differences
is the solver routine used. We will provide a separate function for each solver.

Example files:
```{r}
list.files(extdata.path, pattern = "^uvspec-disort-")
```

For reading this simple output, no special function is needed. We can use `read.table` from base R. Here we read a file with two columns with wavelengths and global spectral energy irradiance (named "eglo" in libRadtran) in $mW\,m^{-2}\,nm^{-1}$. The file was created with one of the 'uvspec' examples included with libRadtran, but reducing the output to two columns.

The first few lines of the file look like this:
```
  250.000  0.000000e+00
  251.000  0.000000e+00
  252.000  0.000000e+00
  253.000  0.000000e+00
...
```

```{r}
uvspec.2col.file <- 
  system.file("extdata", "uvspec-plain-2col.dat", 
              package = "photobiologyInOut", mustWork = TRUE)
lrt.df <- read.table(file = uvspec.2col.file,
                     col.names = c("w.length", "s.e.irrad"))
uvspec.01.spct <- source_spct(w.length = lrt.df$w.length,
                               s.e.irrad = lrt.df$s.e.irrad * 1e-3)
summary(uvspec.01.spct)
cat(comment(uvspec.01.spct))
autoplot(uvspec.01.spct, range = c(250, 2500), unit.out = "photon")
```

An example using solver `disort` and our function read_uvspec_disort() follows.

The first few lines of the file look like this:
```
  290.000  0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00
  291.000  1.046525e-11  4.800521e-06  1.674293e-07  2.374267e-12  5.342789e-07  2.664720e-08
  292.000  5.888299e-10  2.813865e-05  9.814190e-07  1.335888e-10  3.162808e-06  1.561977e-07
  293.000  4.383296e-09  5.764524e-05  2.010660e-06  9.944455e-10  6.522616e-06  3.200065e-07
...
```

```{r}
uvspec.disort.file <- 
  system.file("extdata", "uvspec-disort.dat", 
              package = "photobiologyInOut", mustWork = TRUE)
uvspec.02.spct <- read_uvspec_disort(uvspec.disort.file)
summary(uvspec.02.spct)
cat(comment(uvspec.02.spct))
autoplot(uvspec.02.spct, unit.out = "photon")
```

The data contains also estimates of diffuse and direct spectral irradiance. Here we plot the total energy irradiance with a solid line and the diffuse component with a dashed line.

```{r}
ggplot(uvspec.02.spct) +
  geom_line() +
  geom_line(aes(y = s.e.irrad.diff), linetype = "dashed")
```

The uvspec file used to generate the spectrum read above is:
```
data_files_path uvspec_home/data/
atmosphere_file uvspec_home/data/atmmod/afglms.dat
source solar uvspec_home/data/solar_flux/kurudz_1.0nm.dat
rte_solver disort
mol_abs_param lowtran
deltam on
number_of_streams 6
wavelength 290 900
day_of_year 287
altitude 0.012
albedo_library IGBP
brdf_rpv_type 5
mol_modify O3 288 DU
mol_modify H2O 10 MM
sza 69.4662
sur_temperature 273
```

If we plan to save and reuse the spectral object, it is recommended to append the input file to the comment.

```{r}
uvspec.disort.inp.file <- 
  system.file("extdata", "uvspec-disort.inp", 
              package = "photobiologyInOut", mustWork = TRUE)
comment(uvspec.02.spct) <- paste(comment(uvspec.02.spct),
                                 read_file(uvspec.disort.inp.file),
                                 sep = "\n\n")
cat(comment(uvspec.02.spct))
```

We give two additional examples, which will most likely need some adjustment by users, as these are for output from libRadtran post-processed to add additional information. These are included in the package because myself and collaborators use these formats heavily. In fact users could develop shell scripts or Perl scripts using the same output format.

### Output enriched with time and date data

In this case the file to be read is similar as above, but including separate columns for direct and diffuse components of the spectral energy irradiance. In addition two columns, one with date strings in ISO format and
one with times have been added. The file instead of containing a single spectrum, contains several spectra in long form.

Example files:
```{r}
list.files(extdata.path, pattern = "^uvspec-multi")
```

The first few lines of the file look like this:
```
290.000 2015-05-19 11_00_00 0.000000e+00 0.000000e+00
291.000 2015-05-19 11_00_00 0.000000e+00 0.000000e+00
292.000 2015-05-19 11_00_00 0.000000e+00 0.000000e+00
293.000 2015-05-19 11_00_00 1.893645e-05 3.439497e-05
294.000 2015-05-19 11_00_00 1.648530e-04 2.764368e-04
...
```

A function is included for reading data saved in a text file in this format. It also automatically converts $mW\,m^{-2}\,nm^{-1}$ into $W\,m^{-2}\,nm^{-1}$.

```{r}
uvspec.multi.file <- 
  system.file("extdata", "uvspec-multi.dat", 
              package = "photobiologyInOut", mustWork = TRUE)
lbr.multi.spct <- read_uvspec_disort_vesa(uvspec.multi.file)
print(lbr.multi.spct, n = 5)
```

### Scripts developed by Anders Lindfors

A model for the simulation of the solar spectrum was developed at the Finnish
Meteorological Institute (FMI) by Dr.\ Anders Lindfors and collaborators and
uses functions from 'libRadtran' as its engine, but saves some additional
metadata to the output file. The main addition is related to the estimation
of the effect of clouds.

Example files:
```{r}
list.files(extdata.path, pattern = "^fmi-al-")
```

Functions `read_fmi_cum()` and `read_m_fmi_cum()` can be used to read text files
output as daily integrated spectral irradiance. In other words cummulated
daily data. Function `read_fmi2mspct()` reads spectral irradiance, extracting
multiple sequential spectra from a single file.

The first few lines of the files with cummulated data look like this:
```
# date number_of_scans start_scan stop_scan max_time_gap max_sza_gap warnings
# 20140821 15 3:30:00 17:30:00 60 7.4
# wavelength exposure(J/m2/nm)
2900 0.00000000e+00
2910 2.93132235e-05
2920 7.23526379e-04
...
```

We can read an individual file into a `source_spct` object while adding
some metadata read from the file header.
In this case values are for daily global spectral energy exposures rather
than irradiances. Wavelengths are expressed in Angstroms instead of nanometres.

```{r}
fmi.file <- 
  system.file("extdata", "fmi-al-2014-08-21_cum.hel", 
              package = "photobiologyInOut", mustWork = TRUE)
z.spct <- read_fmi_cum(fmi.file)
class_spct(z.spct)
when_measured(z.spct)
z.spct
```

With function `read_m_fmi_cum` with an ``m'' in the name we can read several files each containing a single spectrum. The returned object is a collection of source spectra.

```{r}
fmi.files <- 
  system.file("extdata", c("fmi-al-2014-08-21_cum.hel", "2014-08-21_cum.hel"),
              package = "photobiologyInOut", mustWork = TRUE)
z.mspct <- read_m_fmi_cum(fmi.files)
class(z.mspct)
when_measured(z.mspct)
z.mspct
```

Above we gave the names of the files explicitly, but as we show here, one can build on-the-fly a list of file names matching some pattern. The example below is not run, as the location of example files may vary. The string `"."` should be replaced with the path to the folder where the files to be read are located.

```{r, eval=FALSE}
fmi.files <- list.files(".", "*cum.hel")
fmi.files <- paste(".", fmi.files, sep = "")
z1.mspct <- read_m_fmi_cum(fmi.files)
class(z1.mspct)
when_measured(z1.mspct)
z1.mspct
```

One also add a geocode at the time of import (or later).

```{r, message=FALSE}
# because of Google's query limits call will frequently fail without a key
# my.geocode <- ggmap::geocode("Kumpula, Helsinki, Finland", source = "google")
my.geocode <- data.frame(lon = 24.96474, lat = 60.20911)
z2.mspct <-
  read_m_fmi_cum(fmi.files,
                 geocode = my.geocode)
class(z2.mspct)
when_measured(z2.mspct)
where_measured(z2.mspct)
z2.mspct
```

Files with spectral irradiance contain data for multiple spectra stored as
text. Each spectrum is delimited at the top by a header line with
metadata and at the end by `"end"` in a line by itself.

The first few and last lines for each spectrum look like this:
```
# 20130501 3:30:00 3:30:00 82.656
210.00 2900 0
210.00 2910 0
210.00 2920 0
...
210.00 8480 103.73018
210.00 8490 104.63495
210.00 8500 90.18384
end
...
```
The number of spectra, range of wavelengths and the length of each spectrum
can vary. Function `read_fmi2mspct()` uses matching to the delimiters to read
all the data in all cases.

```{r}
fmi.file <- 
  system.file("extdata", "fmi-al-2013-05-01.hel", 
              package = "photobiologyInOut", mustWork = TRUE)
z3.mspct <- read_fmi2mspct(fmi.file)
class(z3.mspct)[1:2]
when_measured(z3.mspct[[1]])
length(z3.mspct)
names(z3.mspct)
when_measured(z3.mspct[[1]])
what_measured(z3.mspct[[1]])
```

## Online public repositories of spectral data

Functions for importing spectral data downloaded from repositories.

| R function             | Data repository        | Version           | `class` of value |
|:-----------------------|:-----------------------|:------------------|:-----------------|
| read_FReD_csv()      | Floral Reflectance db. | 2017-03-19        | reflector_spct  |
| read_ASTER_txt()     | ASTER spectral lib.    | version 2.0 ASCII | reflector_spct  |

### FReD Floral Reflectance Database

Example files:
```{r}
list.files(extdata.path, pattern = "^FReD")
```

The files downloaded from FReD do not contain a header, but the first column
indicates the flower ID.

```
157, 300, 0.0627119 
157, 301, 0.0654036 
157, 302, 0.0677941 
157, 303, 0.0670396 
...
```

```{r}
fred.file <- 
  system.file("extdata", "FReDflowerID_157.csv", 
              package = "photobiologyInOut", mustWork = TRUE)
fred.spct <- read_FReD_csv(file = fred.file, 
                           label = "Gazania heterochaeta",
                           geocode = data.frame(lat = -28.8751, lon = 17.2293))
```

In this case as there is no metadata present in the file, it needs to be supplied 
by the user.

```{r}
fred.spct
cat(comment(fred.spct))
autoplot(fred.spct)
```

### ASTER spectral database

The files downloaded from ASTER contain a 25-lines-long header, but at the 
moment only the first field is decoded, as the whole header copied as a 
comment.

```
Name: Dry grass
Type:  Vegetation
Class:  Grasses
Subclass:  Dry grass
Particle Size:  Solid
Sample No.:  drygrass.doc
Owner:  JHU
Wavelength Range:  All
Origin:  The entire spectral range was measured at Johns Hopkins University.

Description:  Dry grass.  Spectra were assembled from two segments; the 
bidirectional VNIR and SWIR comprising segment one, and the hemispherical 
MWIR and TIR comprising segment two. The VNIR/SWIR spectrum was 
measured in the laboratory at JHU with a GER IRIS Mark IV, using a large piece 
of sod.  The grass was illuminated from directly above and measured at a 
reflectance angle of 60 degrees to avoid viewing the thatch. 
Measurement:  Bidirectional and directional hemispherical reflectance. 
First Column:  X
Second Column:  Y  
X Units:  Wavelength (micrometers)
Y Units:  Reflectance (percent)
First X Value: 0.38049
Last X Value: 14.011 
Number of X Values: 2559
Additional Information:  None.
 
0.38049	14.249	 
0.38299	14.251		
0.38544	14.546		
0.38791	14.694		
...
```

```{r}
aster.file <- 
  system.file("extdata", "aster-Rpc-drygrass.txt", 
              package = "photobiologyInOut", mustWork = TRUE)
aster.spct <- read_ASTER_txt(file = aster.file)
```

The label and comment are set from the file header.

```{r}
aster.spct
cat(comment(aster.spct))
autoplot(aster.spct)
```

## Other R packages

A general way of exchanging data with other R packages or for use with base R functions is to create a matrix from a collection of spectra with `as.matrix()`, or a collection of spectra from a matrix with one of the `as.xxxx_mspct()` methods such as `as.source_spct`. Such methods are defined in package 'photobiology' as well as method `join_mspct()` for conversion of collections of spectra into _wide_ data frames. However, a matrix is only guaranteed to contain numeric data and a `"dim"` attribute, while conversion to a data frame preserves only part of the metadata. These generic conversions cannot be reversed without loss of information. When possible use the package specific functions as they automate much of the recovery and preservation of metadata.

Functions for exchanging data with _foreign_ R packages.

| R function             | Foreign R package     | Function          | `class` of value |
|:-----------------------|:-----------------------|:------------------|:-----------------|
| hyperSpec2spct()      | ’hyperSpec’            | import            | source_spct    |
| spct2hyperSpec()      | ’hyperSpec’            | export            | hyperSpec        |
| hyperSpec2mspct()      | ’hyperSpec’            | import            | source_mspct    |
| mspct2hyperSpec()      | ’hyperSpec’            | export            | hyperSpec        |
| colorSpec2spct()      | ’colorSpec’            | import            | source_spct    |
| spct2colorSpec()      | ’colorSpec’            | export            | colorSpec        |
| colorSpec2mspct()      | ’colorSpec’            | import            | source_mspct    |
| mspct2colorSpec()      | ’colorSpec’            | export            | colorSpec        |
| chroma_spct2colorSpec()      | ’colorSpec’            | export            | colorSpec        |
| rspec2mspct()          | ’pavo’                 | import            | source_mspct    |


### To 'hyperSpec'

Can export to `''hyperSpec''` objects only collections of spectra where all members have identical `w.length` vectors, as objects of class `hyperSpec` store a single vector of wavelengths for the whole collection of spectra.

```{r, eval=eval_hyperSpec}
z2.hspct <- mspct2hyperSpec(z2.mspct, "s.e.irrad")
class(z2.hspct)
# plot(z2.hspct)
```

### From 'hyperSpec'

Can import only data with wavelength in nanometres. Other quantities and units are not supported by the 'photobiology' classes for spectral data. See package 'hyperSpec' vignette "laser" for details on the data and the conversion of the original wavelength units into nanometres.

```{r, eval=eval_hyperSpec}
data(laser)
class(laser)
laser
plot(laser)
```

We assume here, that the quantity for the spectral emission of the laser is
spectral _energy_ irradiance, expressed in $mW\,m^{-2}\,nm^{-1}$. This is likely to be
wrong but for the sake of showing how the conversion takes place is irrelevant.
The parameter `multiplier` can be passed a numeric argument to rescale
the original data. The default multiplier is 1.

```{r, eval=eval_hyperSpec}
wl(laser) <- list (
  wl = 1e7 / (1/405e-7 - wl (laser)),
  label = expression (lambda / nm)
)
laser
plot(laser)
laser.mspct <-
  hyperSpec2mspct(laser, "source_spct", "s.e.irrad", multiplier = 1e-3)
ggplot(laser.mspct[[1]]) +
  geom_line() +
  stat_peaks(geom = "text", vjust = -1, label.fmt = "%.6g nm", color = "red")
```

### From 'colorSpec'

```{r, eval = eval_colorSpec}
# bug that needs to be fixed
fluorescent.mspct <- colorSpec2mspct(colorSpec::Fs.5nm)
print(fluorescent.mspct, n = 3, n.members = 3)
```

```{r, eval = eval_colorSpec}
colorSpec2mspct(colorSpec::Hoya)
```

```{r, eval = eval_colorSpec}
fluorescent.spct <- colorSpec2spct(colorSpec::Fs.5nm)
autoplot(fluorescent.spct, annotations = "")
```

```{r, eval = eval_colorSpec}
colorSpec2chroma_spct(colorSpec::xyz1931.5nm)
```

### To 'colorSpec'

```{r, eval = eval_colorSpec}
sun.cspec <- spct2colorSpec(sun.spct)
plot(sun.cspec, col = "blue")
```

```{r, eval = eval_colorSpec}
spct2colorSpec(yellow_gel.spct)
```

```{r, eval = eval_colorSpec}
chroma_spct2colorSpec(beesxyzCMF.spct)
```
### Wrappers on 'colorSpec' functions.

Functions `spct_CCT()`, `spct_CRI()` and `spct_SSI()` call the respective `compute` functions after converting `source_spct` objects. CCT, CRI and SSI are meaningful for human vision although they are used not only for general illumination but also in photography and cinematography. They can be generalized by overriding the default references from human vision with those of cameras or other visual systems.

```{r}
spct_CCT(white_led.source_spct) # correlated color temperature
spct_CRI(white_led.source_spct) # color rendition index
spct_CRI(white_led.source_spct, named = TRUE)
spct_SSI(white_led.source_spct, sun.spct) # spectral similarity index
```

### From 'pavo'

In this example we convert an `rspec` object from package 'pavo' into a collection of spectra and then we plot it with `ggplot` methods from package `ggspectra' (an extension to `ggplot2'). The data are the spectral reflectance of the plumage from seven different individual birds of the same species, measured in three different body parts.

```{r, eval = eval_pavo}
data(sicalis)
class(sicalis)
names(sicalis)
```

We convert the data into a collection of spectra, and calculate summaries for three spectra.

```{r, eval = eval_pavo}
sicalis.mspct <- rspec2mspct(sicalis, "reflector_spct", "Rpc")
summary(sicalis.mspct[[1]])
summary(sicalis.mspct[[2]])
summary(sicalis.mspct[[3]])
```

We convert the subset of the collection corresponding to the first individual into a single spectra object for plotting with `ggplot`.

```{r, eval = eval_pavo}
ggplot(rbindspct(sicalis.mspct[1:3])) +
  aes(linetype = spct.idx) +
  ylim(0,0.3) +
  geom_line()
```

Here we extract the ``crown'' data from all individuals and plot these spectra in a single plot.

```{r, eval = eval_pavo}
print(sicalis.mspct[c(TRUE, FALSE, FALSE)])
ggplot(rbindspct(sicalis.mspct[c(TRUE, FALSE, FALSE)])) +
  aes(linetype = spct.idx) +
  ylim(0,0.15) +
  geom_line() +
  ggtitle("'crown' reflectance spectra")
```

We calculate the mean reflectance in wavebands corresponding to ISO colors obtaining a data frame. We then add to this returned data frame a factor indicating the body parts.

```{r, eval = eval_pavo}
refl.by.band <- reflectance(sicalis.mspct, w.band = list(Red(), Green(), Blue(), UVA()))
refl.by.band$body.part <- rep(c("crown", "throat", "breast"), 7)
```

```{r, eval = eval_pavo}
refl.red <- reflectance(sicalis.mspct, w.band = Red())
names(refl.red)[2] <- "red.reflectance"
refl.red$body.part <- rep(c("crown", "throat", "breast"), 7)
ggplot(refl.red, aes(x = body.part, y = red.reflectance)) +
  stat_summary(fun.data = "mean_se", color = "red") +
  geom_point(alpha = 0.5)
```

## Dealing with odd and bad data

### Using locales

Most functions in this package have a parameter `locale`, that accepts `readr::locale` objects as arguments. At the moment only the time zone and decimal mark are respected. This allows files using comma for decimal marker be easily imported, or the dates and times **in the input file** be interpreted in a given time zone. Setting the correct time zone is very important to avoid errors. Time coordinates are always stored in the created objects using universal time coordinates ("UTC").

```{r}
jaz.irrad.comma.file <- 
  system.file("extdata", "oo-spectrum-comma.jazIrrad", 
              package = "photobiologyInOut", mustWork = TRUE)
my.locale <- locale(decimal_mark = ",", tz = "Europe/Helsinki")
jaz00.spct <- read_oo_jazirrad(file = jaz.irrad.comma.file,
                               locale = my.locale)
```

```{r}
jaz00.spct
```

### Overriding default metadata

We revisit now the Jaz irradiance data to show how the metadata can be changed by the user if needed (e.g. clock settings at the time of data acquisition were wrong).

A variable with the user supplied date and time data, or the date read from
the header (the text itself) not the file date as the file date may not reflect
the creation date and time.

```{r}
jaz.s.irrad.file <- 
  system.file("extdata", "oo-spectrum.jazIrrad", 
              package = "photobiologyInOut", mustWork = TRUE)
```

```{r, warning=FALSE}
jaz01.spct <- read_oo_jazirrad(file = jaz.s.irrad.file,
                               date = NULL)
when_measured(jaz01.spct)
```

```{r, warning=FALSE}
jaz02.spct <- read_oo_jazirrad(file = jaz.s.irrad.file,
                               date = ymd_hms("2015-11-15 12:00:00"))
when_measured(jaz02.spct)
```

```{r, warning=FALSE}
jaz03.spct <- read_oo_jazirrad(file = jaz.s.irrad.file,
                               date = now())
when_measured(jaz03.spct)
```

### Adding additional metadata

When can add a geocode, either directly by giving latitude and longitude coordinates or by generating it from a Google maps search using function `ggmap::geocode()` as shown here.

```{r, message=FALSE,warning=FALSE}
my.geocode <- data.frame(lon = 25.02006, lat = 60.22525)
jaz04.spct <- read_oo_jazirrad(file = jaz.s.irrad.file,
                               geocode = my.geocode)
jaz04.spct
where_measured(jaz04.spct)
```