Skip to content

Conversation

@HIT-cwh
Copy link
Collaborator

@HIT-cwh HIT-cwh commented Apr 24, 2024

Usage

from mmengine.config import ConfigDict
from xtuner.registry import BUILDER
from xtuner.dataset.json_dataset import load_json_file

# Case 1 : load one json file
cfg = ConfigDict(type=load_json_file, data_files='/path/to/your/json.json')
ds = BUILDER.build(cfg)

# Case 2 : load several json files
cfg = ConfigDict(type=load_json_file, data_files=['/path/to/your/json1.json', 
                                                  '/path/to/your/json2.json'])
ds = BUILDER.build(cfg)

# Case 3: load all the files in specific dir
cfg = ConfigDict(type=load_json_file, data_dir='/path/to/your/dir')
ds = BUILDER.build(cfg)

# Case 4: load all the files in specific dir whose suffix is `specific_suffix`
cfg = ConfigDict(type=load_json_file, data_dir='/path/to/your/dir', suffix='.json.cot')
ds = BUILDER.build(cfg)
@pppppM pppppM merged commit 649cab9 into InternLM:main Apr 24, 2024
@HIT-cwh HIT-cwh deleted the support_json_load branch April 25, 2024 06:33
llkn-2 pushed a commit to llkn-2/xtuner that referenced this pull request Jul 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

2 participants