mirror of
https://github.com/nushell/nushell
synced 2024-12-29 06:23:11 +00:00
f879c00f9d
# Description
There are times where explicitly specifying a schema for a dataframe is
needed such as:
- Opening CSV and JSON lines files and needing provide more information
to polars to keep it from failing or in a desire to override default
type conversion
- When converting a nushell value to a dataframe and wanting to override
the default conversion behaviors.
This pull requests provides:
- A flag to allow specifying a schema when using dfr into-df
- A flag to allow specifying a schema when using dfr open that works for
CSV and JSON types
- A new command `dfr schema` which displays schema information and will
allow display support schema dtypes
Schema is specified creating a record that has the key value and the
dtype. Examples usages:
```
{a:1, b:{a:2}} | dfr into-df -s {a: u8, b: {a: i32}} | dfr schema
{a: 1, b: {a: [1 2 3]}, c: [a b c]} | dfr into-df -s {a: u8, b: {a: list<u64>}, c: list<str>} | dfr schema
dfr open -s {pid: i32, ppid: i32, name: str, status: str, cpu: f64, mem: i64, virtual: i64} /tmp/ps.jsonl | dfr schema
```
Supported dtypes:
null
bool
u8
u16
u32
u64
i8
i16
i32
i64
f32
f64
str
binary
date
datetime[time_unit: (ms, us, ns) timezone (optional)]
duration[time_unit: (ms, us, ns)]
time
object
unknown
list[dtype]
structs are also supported but are specified via another record:
{a: u8, b: {d: str}}
Another feature with the dfr schema command is that it returns the data
back in a format that can be passed to provide a valid schema that can
be passed in as schema argument:
<img width="638" alt="Screenshot 2024-01-29 at 10 23 58"
src="https://github.com/nushell/nushell/assets/56345/b49c3bff-5cda-4c86-975a-dfd91d991373">
---------
Co-authored-by: Jack Wright <jack.wright@disqo.com>
|
||
|---|---|---|
| .. | ||
| nu-cli | ||
| nu-cmd-base | ||
| nu-cmd-dataframe | ||
| nu-cmd-extra | ||
| nu-cmd-lang | ||
| nu-color-config | ||
| nu-command | ||
| nu-engine | ||
| nu-explore | ||
| nu-glob | ||
| nu-json | ||
| nu-lsp | ||
| nu-parser | ||
| nu-path | ||
| nu-plugin | ||
| nu-pretty-hex | ||
| nu-protocol | ||
| nu-std | ||
| nu-system | ||
| nu-table | ||
| nu-term-grid | ||
| nu-test-support | ||
| nu-utils | ||
| nu_plugin_custom_values | ||
| nu_plugin_example | ||
| nu_plugin_formats | ||
| nu_plugin_gstat | ||
| nu_plugin_inc | ||
| nu_plugin_python | ||
| nu_plugin_query | ||
| README.md | ||
Nushell core libraries and plugins
These sub-crates form both the foundation for Nu and a set of plugins which extend Nu with additional functionality.
Foundational libraries are split into two kinds of crates:
- Core crates - those crates that work together to build the Nushell language engine
- Support crates - a set of crates that support the engine with additional features like JSON support, ANSI support, and more.
Plugins are likewise also split into two types:
- Core plugins - plugins that provide part of the default experience of Nu, including access to the system properties, processes, and web-connectivity features.
- Extra plugins - these plugins run a wide range of different capabilities like working with different file types, charting, viewing binary data, and more.