summaryrefslogtreecommitdiff
path: root/.circleci
AgeCommit message (Collapse)Author
2024-01-31Enable io_uring in stress test (#12313)Akanksha Mahajan
Summary: Enable io_uring in stress test Pull Request resolved: https://github.com/facebook/rocksdb/pull/12313 Test Plan: Crash test Reviewed By: anand1976 Differential Revision: D53238319 Pulled By: akankshamahajan15 fbshipit-source-id: c0c8e6a6479f6977210370606e9d551c1299ba62
2024-01-19Use the correct Docker Image for RocksJava on Linux (#12169)Adam Retter
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/12169 Reviewed By: pdillinger Differential Revision: D52715225 Pulled By: ajkr fbshipit-source-id: 28476d363034fa1bb9c8c919d577c03b6391451b
2023-12-21Initial CircleCI -> GitHub Actions migration (#12163)Peter Dillinger
Summary: * Largely based on https://github.com/facebook/rocksdb/issues/12085 but grouped into one large workflow because of bad GHA UI design (see comments). * Windows job details consolidated into an action file so that those jobs can easily move between per-pr-push and nightly. * Simplify some handling of "CIRCLECI" environment and add "GITHUB_ACTIONS" in the same places * For jobs that we want to go in pr-jobs or nightly there are disabled "candidate" workflows with draft versions of those jobs. * ARM jobs are disabled waiting on full GHA support. * build-linux-java-static needed some special attention to work, due to GLIBC compatibility issues (see comments). Pull Request resolved: https://github.com/facebook/rocksdb/pull/12163 Test Plan: Nightly jobs can be seen passing between these two links: https://github.com/facebook/rocksdb/actions/runs/7266835435/job/19799390061?pr=12163 https://github.com/facebook/rocksdb/actions/runs/7269697823/job/19807724471?pr=12163 And per-PR jobs of course passing on this PR. Reviewed By: hx235 Differential Revision: D52335810 Pulled By: pdillinger fbshipit-source-id: bbb95196f33eabad8cddf3c6b52f4413c80e034d
2023-11-22Convert all but one windows job to nightly (#12089)Peter Dillinger
Summary: ... because they are expensive and rarely disagree with each other. Historical data indicates that the 2019 job is most sensitive to failure. https://fburl.com/scuba/opensource_ci_jobs/ntq3ue3p https://fburl.com/scuba/opensource_ci_jobs/0xo91j5f Pull Request resolved: https://github.com/facebook/rocksdb/pull/12089 Test Plan: CI Reviewed By: ajkr Differential Revision: D51530386 Pulled By: pdillinger fbshipit-source-id: 8b676d6e01096e359a0f465b59d81ac10f4f7969
2023-11-14Build RocksDBJava on Windows with Java8. (#12068)Radek Hubner
Summary: At the moment RocksDBJava uses the default CIrcleCI JVM on Windows builds. This can and has changed in the past and can cause some incompatibilities. This PR addresses the problem of explicitly installing and using Liberica JDK 8 as Java 8 Is the primary target for RocksdbJava. Pull Request resolved: https://github.com/facebook/rocksdb/pull/12068 Reviewed By: cbi42 Differential Revision: D51307233 Pulled By: ajkr fbshipit-source-id: 9cb4e173d8a9ac42e5f9fda1daf012302942fdbc
2023-10-31Fix the ZStd checksum (#12005)Adam Retter
Summary: Somehow we had the wrong checksum when validating the ZStd 1.5.5 download for RocksJava in the previous Pull Request - https://github.com/facebook/rocksdb/pull/9304. This PR fixes that. Pull Request resolved: https://github.com/facebook/rocksdb/pull/12005 Reviewed By: jaykorean Differential Revision: D50840338 Pulled By: cbi42 fbshipit-source-id: 8a92779d3bef013d812eecb89aaaf33fc73991ec
2023-10-30Remove build dependencies for java tests. (#12021)Radek Hubner
Summary: Final fix for https://github.com/facebook/rocksdb/issues/12013 - Reverting back changes on CirleCI explicit image declaration. - Removed CMake dependencies between java classed and java test classes. Pull Request resolved: https://github.com/facebook/rocksdb/pull/12021 Reviewed By: akankshamahajan15 Differential Revision: D50745392 Pulled By: ajkr fbshipit-source-id: 6a7a1da1e7e4da8da72130c9272915974e10fffc
2023-10-25Downgrade windows 2019 build to older image. (#12014)Radek Hubner
Summary: This should fix failed java windows build https://github.com/facebook/rocksdb/issues/12013 Pull Request resolved: https://github.com/facebook/rocksdb/pull/12014 Reviewed By: ajkr Differential Revision: D50664503 Pulled By: akankshamahajan15 fbshipit-source-id: 3466ce42d3cae3f8e0beba88a18744d647a32a2c
2023-10-24Fix windows build errors (rdtsc and fnptr) (#12008)Peter Dillinger
Summary: Combining best parts of https://github.com/facebook/rocksdb/issues/11794 and https://github.com/facebook/rocksdb/issues/11766, fixing the CircleCI config in the latter. I was going to amend the latter but wasn't granted access. Ideally this would be ported back to 8.4 branch and crc32c part into 8.3 branch. Pull Request resolved: https://github.com/facebook/rocksdb/pull/12008 Test Plan: CI Reviewed By: hx235 Differential Revision: D50616172 Pulled By: pdillinger fbshipit-source-id: fa7f778bc281e881a140522e774f480c6d1e5f48
2023-10-18Add RocksJava tests to CMake (#11756)Radek Hubner
Summary: Refactor CMake build to allow run java tests via CMake, including Windows. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11756 Reviewed By: hx235 Differential Revision: D50370739 Pulled By: akankshamahajan15 fbshipit-source-id: ae05cc08a0f9bb2a0a4f1ece02c523fb465bb817
2023-10-17Perform java static checks in CI (#11221)Alan Paxton
Summary: Integrate pmd on the Java API to catch and report common Java coding problems; fix or suppress a basic set of PMD checks. Link pmd into java build / CI Add a pmd dependency to maven Add a jpmd target to Makefile which runs pmd Add a workflow to Circle CI which runs pmd Configure an initial default pmd for CI Repair lots of very simple PMD reports generated when we apply pmd-rules.xml Repair or exception for PMD rules in the CloseResource category, which finds unclosed AutoCloseable resources. We special-case the configuration of CloseResource and use the // NOPMD comment in source the avoid reports where we are the API creating an AutoCloseable, and therefore returning an unclosed resource is correct behaviour. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11221 Reviewed By: akankshamahajan15 Differential Revision: D50369930 Pulled By: jowlyzhang fbshipit-source-id: a41c36b44b3bab7644df3e9cc16afbdf33b84f6b
2023-08-21Circleci macos sunset (#11633)Levi Tamasi
Summary: [draft] this PR is created in order to test CI changes Closes: https://github.com/facebook/rocksdb/pull/11543 Pull Request resolved: https://github.com/facebook/rocksdb/pull/11633 Reviewed By: akankshamahajan15 Differential Revision: D48525552 Pulled By: cbi42 fbshipit-source-id: 758d57f248304213228af459789459cc2f0bf419
2023-02-21Revert enabling IO uring in db_stress (#11242)anand76
Summary: IO uring usage is causing crash test failures due to bad cqe data being returned in the uring. Revert the change to enable IO uring in db_stress, and also re-enable async_io in CircleCI so that code path can be tested. Added the -use_io_uring flag to db_stress that, when false, will wrap the default env in db_stress to emulate async IO. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11242 Reviewed By: akankshamahajan15 Differential Revision: D43470569 Pulled By: anand1976 fbshipit-source-id: 7c69ac3f53a79ade31d37313f815f1a4b6108b75
2023-02-16Return NotSupported in scan if IOUring not supported and enable IOUring in ↵akankshamahajan
db_stress for async_io testing (#11197) Summary: - Return NotSupported in scan if IOUring not supported if async_io is enabled - Enable IOUring in db_stress for async_io testing - Disable async_io in circleci crash testing as circleci doesn't support IOUring Pull Request resolved: https://github.com/facebook/rocksdb/pull/11197 Test Plan: CircleCI jobs Reviewed By: anand1976 Differential Revision: D43096313 Pulled By: akankshamahajan15 fbshipit-source-id: c2c53a87636950c0243038b9f5bd0d91608e4fda
2023-02-08Mitigate presumed OOM in CircleCI (#11206)Peter Dillinger
Summary: We've seen many instances of build-linux-static_lib-alt_namespace-status_checked failing like this: ``` g++: fatal error: Killed signal terminated program cc1plus compilation terminated. make: *** [Makefile:2507: utilities/transactions/transaction_test.o] Error 1 ``` It's understandable that so many static linking jobs could exhaust memory. The executor only has 16 vcores, so going from 32 down to 24 shouldn't hurt build time. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11206 Test Plan: will watch CI Reviewed By: ajkr Differential Revision: D43137246 Pulled By: pdillinger fbshipit-source-id: 050b0f700c285dd913bcae8b4a76a44d04bb0356
2023-02-07Revert to LIB_MODE=static for optimized builds (#11195)Peter Dillinger
Summary: Continuous performance testing indicates there's a small performance hit with shared library (-fPIC) builds, so while retaining the motivation for https://github.com/facebook/rocksdb/issues/11168, we set the default for DEBUG_LEVEL=0 Makefile builds back to LIB_MODE=static. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11195 Test Plan: CI, with some updated checks and removal of some now obsolete LIB_MODE overrides Reviewed By: cbi42 Differential Revision: D43090576 Pulled By: pdillinger fbshipit-source-id: 755fe5d07005f85caf24e16f90228ffd46a6e250
2023-02-06Fix more CircleCI jobs for LIB_MODE=shared default (#11193)Peter Dillinger
Summary: There are a set of jobs using libbenchmark that have linker failures with new default LIB_MODE=shared. This change adds build-linux-run-microbench to the set using LIB_MODE=static to work around the linker failures. I haven't dug into how to fix them. There is another set of jobs using folly that have linker failures with new default LIB_MODE=shared. I tried fixing these by adding --shared-libs to the folly build, but that doesn't work. It kinda looks like the folly shared libs build is simply broken with the boost dependency: ``` /usr/bin/ld: /tmp/fbcode_builder_getdeps-ZrootZprojectZthird-partyZfollyZbuildZfbcode_builder-root/installed/boost-Z1Z72zV-c0-0f3HkylpzONnr1dsHYDaR2GyTLzYdkck/lib/libboost_filesystem.a(exception.o): relocation R_X86_64_PC32 against symbol `_ZTVN5boost10filesystem16filesystem_errorE' can not be used when making a shared object; recompile with -fPIC /usr/bin/ld: final link failed: bad value collect2: error: ld returned 1 exit status ``` I tried updating folly to the latest commit and that didn't help. Otherwise, I didn't dig deeper into fixing that so have added build-linux-clang-13-asan-ubsan-with-folly to the set using LIB_MODE=static Also since I saw a flaky failure (not the first time), increased the timeout on build-linux-unity-and-headers job. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11193 Test Plan: CI Reviewed By: cbi42 Differential Revision: D43061203 Pulled By: pdillinger fbshipit-source-id: c641671f93087f0214ea261ea895bccf657cb1a9
2023-02-06CI Benchmarking. Reduce runtime further as overhead appears to have risen. ↵Alan Paxton
(#11189) Summary: We had miscalculated (not sure if I suddenly can’t count, or if there is something else going on), and need to leave more overhead to get the benchmarks to run reliably under 1 hour. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11189 Reviewed By: cbi42 Differential Revision: D43052045 Pulled By: ajkr fbshipit-source-id: 3fe68432ed76a1f87d34129b0246e6b6a70a49f2
2023-02-03Use LIB_MODE=shared build by default with make (#11168)Peter Dillinger
Summary: With https://github.com/facebook/rocksdb/issues/11150 this becomes a practical change that I think is overall good for developer efficiency. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11168 Test Plan: More efficient build of all unit tests and tools: ``` $ git clean -fdx $ du -sh . 522M . $ /usr/bin/time make -j32 LIB_MODE=static ... 14270.63user 1043.33system 11:19.85elapsed 2252%CPU (0avgtext+0avgdata 1929944maxresident)k ... $ du -sh . 62G . $ ``` Vs. ``` $ git clean -fdx $ du -sh . 522M . $ /usr/bin/time make -j32 LIB_MODE=shared ... 9479.87user 478.26system 7:20.82elapsed 2258%CPU (0avgtext+0avgdata 1929272maxresident)k ... $ du -sh . 5.4G . $ ``` So 1/3 less build time and >90% less space usage. Individual unit test edit-compile-run is not too different. Modifying an average unit test source file: ``` $ touch db/version_builder_test.cc $ /usr/bin/time make -j32 LIB_MODE=static version_builder_test ... 34.74user 3.37system 0:38.29elapsed 99%CPU (0avgtext+0avgdata 945520maxresident)k ``` Vs. ``` $ touch db/version_builder_test.cc $ /usr/bin/time make -j32 LIB_MODE=shared version_builder_test ... 116.26user 43.91system 0:28.65elapsed 559%CPU (0avgtext+0avgdata 675160maxresident)k ``` A little faster with shared. However, modifying an average DB implementation file has an extra linking step with shared lib: ``` $ touch db/db_impl/db_impl_files.cc $ /usr/bin/time make -j32 LIB_MODE=static version_builder_test ... 33.17user 5.13system 0:39.70elapsed 96%CPU (0avgtext+0avgdata 945544maxresident)k ``` Vs. ``` $ touch db/db_impl/db_impl_files.cc $ /usr/bin/time make -j32 LIB_MODE=shared version_builder_test ... 40.80user 4.66system 0:45.54elapsed 99%CPU (0avgtext+0avgdata 1056340maxresident)k ``` A little slower with shared. On the whole, should be faster and lighter weight because of the many unit test files case Reviewed By: cbi42 Differential Revision: D42894004 Pulled By: pdillinger fbshipit-source-id: 9e827e52ace79b86f849b6a24466e318b4b605a7
2023-02-02Remove NUMA setting for benchmark-linux (#11180)changyubi
Summary: benchmark-linux is failing on main branch after https://github.com/facebook/rocksdb/issues/11074 with the following error msg: ``` /usr/bin/time -f '%e %U %S' -o /tmp/benchmark-results/8.0.0/benchmark_overwriteandwait.t1.s0.log.time numactl --interleave=all timeout 1200 ./db_bench --benchmarks=overwrite,waitforcompaction,stats --use_existing_db=1 --sync=0 --level0_file_num_compaction_trigger=4 --level0_slowdown_writes_trigger=20 --level0_stop_writes_trigger=30 --max_background_jobs=4 --max_write_buffer_number=8 --undefok=use_blob_cache,use_shared_block_and_blob_cache,blob_cache_size,blob_cache_numshardbits,prepopulate_blob_cache,multiread_batched,cache_low_pri_pool_ratio,prepopulate_block_cache --db=/tmp/rocksdb-benchmark-datadir --wal_dir=/tmp/rocksdb-benchmark-datadir --num=20000000 --key_size=20 --value_size=400 --block_size=8192 --cache_size=10737418240 --cache_numshardbits=6 --compression_max_dict_bytes=0 --compression_ratio=0.5 --compression_type=none --bytes_per_sync=1048576 --cache_index_and_filter_blocks=1 --cache_high_pri_pool_ratio=0.5 --cache_low_pri_pool_ratio=0 --benchmark_write_rate_limit=0 --write_buffer_size=16777216 --target_file_size_base=16777216 --max_bytes_for_level_base=67108864 --verify_checksum=1 --delete_obsolete_files_period_micros=62914560 --max_bytes_for_level_multiplier=8 --statistics=0 --stats_per_interval=1 --stats_interval_seconds=20 --report_interval_seconds=1 --histogram=1 --memtablerep=skip_list --bloom_bits=10 --open_files=-1 --subcompactions=1 --compaction_style=0 --num_levels=8 --min_level_to_compress=3 --level_compaction_dynamic_level_bytes=true --pin_l0_filter_and_index_blocks_in_cache=1 --duration=600 --threads=1 --merge_operator="put" --seed=1675372532 --report_file=/tmp/benchmark-results/8.0.0/benchmark_overwriteandwait.t1.s0.log.r.csv 2>&1 | tee -a /tmp/benchmark-results/8.0.0/benchmark_overwriteandwait.t1.s0.log /usr/bin/time: cannot run numactl: No such file or directory ``` This PR removes the newly added NUMA setting. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11180 Test Plan: check next main branch run for benchmark-linux Reviewed By: ajkr Differential Revision: D42975930 Pulled By: cbi42 fbshipit-source-id: f084d39aeba9877c0752502e879c5e612b507653
2023-02-02CI Benchmarking. Small configuration changes based on performance analysis. ↵Alan Paxton
(#11074) Summary: First, we made a small reduction in DURATION_RW as runs were exceeding 1 hour and colliding with subsequent runs. See Mark Callaghan’s blog post at http://smalldatum.blogspot.com/2023/01/variance-in-rocksdb-benchmarks-on-cloud.html Configuration parameters which are not consistent with the following email from Mark (see the blog post for more context) have been updated. Where Mark has defined the parameter and we haven't, we define it explicitly. We will need to further monitor for an expected reduction in variance of test times: To match what I did: --- nsecs=1800 dbdir=/data/m/rx resultdir=bm.lc.nt1.cm1.d0 env WRITE_BUFFER_SIZE_MB=16 TARGET_FILE_SIZE_BASE_MB=16 MAX_BYTES_FOR_LEVEL_BASE_MB=64 MAX_BACKGROUND_JOBS=4 NUM_KEYS=20000000 CACHE_SIZE_MB=10240 DURATION_RW=$nsecs DURATION_RO=$nsecs MB_WRITE_PER_SEC=2 NUM_THREADS=1 COMPRESSION_TYPE=none CACHE_INDEX_AND_FILTER_BLOCKS=1 VALUE_SIZE=400 NUMA=1 MIN_LEVEL_TO_COMPRESS=3 COMPACTION_STYLE=leveled bash benchmark_compare.sh $dbdir $resultdir 7.8.fb env WRITE_BUFFER_SIZE_MB=16 TARGET_FILE_SIZE_BASE_MB=16 MAX_BYTES_FOR_LEVEL_BASE_MB=64 MAX_BACKGROUND_JOBS=4 NUM_KEYS=200000000 CACHE_SIZE_MB=10240 DURATION_RW=$nsecs DURATION_RO=$nsecs MB_WRITE_PER_SEC=2 NUM_THREADS=1 COMPRESSION_TYPE=lz4 CACHE_INDEX_AND_FILTER_BLOCKS=1 VALUE_SIZE=400 NUMA=1 MIN_LEVEL_TO_COMPRESS=3 COMPACTION_STYLE=leveled bash benchmark_compare.sh $dbdir $resultdir 7.8.fb Pull Request resolved: https://github.com/facebook/rocksdb/pull/11074 Reviewed By: ajkr Differential Revision: D42969668 Pulled By: cbi42 fbshipit-source-id: 1ea4e6a3901be4016108f93817eb58f74baac21a
2023-01-27Remove RocksDB LITE (#11147)sdong
Summary: We haven't been actively mantaining RocksDB LITE recently and the size must have been gone up significantly. We are removing the support. Most of changes were done through following comments: unifdef -m -UROCKSDB_LITE `git grep -l ROCKSDB_LITE | egrep '[.](cc|h)'` by Peter Dillinger. Others changes were manually applied to build scripts, CircleCI manifests, ROCKSDB_LITE is used in an expression and file db_stress_test_base.cc. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11147 Test Plan: See CI Reviewed By: pdillinger Differential Revision: D42796341 fbshipit-source-id: 4920e15fc2060c2cd2221330a6d0e5e65d4b7fe2
2022-10-28Upgrade CircleCI Windows Build (#10090)Jay Zhuang
Summary: * Upgrade CircleCI orb from 2.4 to 5.0 * Setup vs2022 build * Use image build-in vs2019 and vs2022 * Remove vs2017 * Remove CMAKE_CXX_STANDARD=20 Pull Request resolved: https://github.com/facebook/rocksdb/pull/10090 Reviewed By: ajkr Differential Revision: D40787942 Pulled By: jay-zhuang fbshipit-source-id: cc74c02a9f28dd784a0ba5502c4bfc9ff1a26d3e
2022-10-17Improve / refactor anonymous mmap capabilities (#10810)Peter Dillinger
Summary: The motivation for this change is a planned feature (related to HyperClockCache) that will depend on a large array that can essentially grow automatically, up to some bound, without the pointer address changing and with guaranteed zero-initialization of the data. Anonymous mmaps provide such functionality, and this change provides an internal API for that. The other existing use of anonymous mmap in RocksDB is for allocating in huge pages. That code and other related Arena code used some awkward non-RAII and pre-C++11 idioms, so I cleaned up much of that as well, with RAII, move semantics, constexpr, etc. More specifcs: * Minimize conditional compilation * Add Windows support for anonymous mmaps * Use std::deque instead of std::vector for more efficient bag Pull Request resolved: https://github.com/facebook/rocksdb/pull/10810 Test Plan: unit test added for new functionality Reviewed By: riversand963 Differential Revision: D40347204 Pulled By: pdillinger fbshipit-source-id: ca83fcc47e50fabf7595069380edd2954f4f879c
2022-09-23CI benchmarks return NUM_KEYS to previous size (#10649)Alan Paxton
Summary: Larger size is necessary to stress levels 2, 3 of LSM tree Pull Request resolved: https://github.com/facebook/rocksdb/pull/10649 Reviewed By: ajkr Differential Revision: D39744515 Pulled By: jay-zhuang fbshipit-source-id: 62ff097bfbfdfc26ff1e6290e1e3b71506b7042c
2022-09-22Use grep instead of obsolete egrep (#10701)Anatol Pomozov
Summary: It fixes "egrep: warning: egrep is obsolescent; using grep -E" warning at the systems with newer gnu grep. https://www.phoronix.com/news/GNU-Grep-3.8-Stop-egrep-fgrep Pull Request resolved: https://github.com/facebook/rocksdb/pull/10701 Reviewed By: ajkr Differential Revision: D39737908 Pulled By: cbi42 fbshipit-source-id: 13f3ccfc1a37d18541d156a6b4c2ba24f6c66589
2022-09-11Build and link libfolly with RocksDB (#10103)anand76
Summary: The current integration with folly requires cherry-picking folly source files to include in RocksDB for external CI builds. Its not scaleable as we depend on more features in folly, such as coroutines. This PR adds a dependency from RocksDB to the folly library when ```USE_FOLLY``` or ```USE_COROUTINES``` are set. We build folly using the build scripts in ```third-party/folly```, relying on it to download and build its dependencies. A new ```Makefile``` target, ```build_folly```, is provided to make building folly easier. A new option, ```USE_FOLLY_LITE``` is added to retain the old model of compiling selected folly sources with RocksDB. This might be useful for short-term development. Pull Request resolved: https://github.com/facebook/rocksdb/pull/10103 Reviewed By: pdillinger Differential Revision: D38426787 Pulled By: anand1976 fbshipit-source-id: 33bc84abd9fdc7e2567749f02aa1b2494eb62b2f
2022-09-05reduce memory usage in CircleCI mini crashtest (#10639)Andrew Kryczka
Summary: Example flake where CircleCI reports memory at 99% and process gets killed with signal 9 (likely OOM): https://app.circleci.com/pipelines/github/facebook/rocksdb/18085/workflows/bdadbfe6-c40f-4ccb-a5db-fc8c4036f20a/jobs/475628 The previous settings of max_key=25000000, column_families=10, and log2_keys_per_lock=2 resulted in 3GB memory usage just for SharedState. The locks alone consume at least (25000000 keys per CF) * (10 CFs) / (2^2 keys per lock) * (40 bytes per lock) = 2.3GB. This PR reduces it 10x by reducing max_key by that factor. Pull Request resolved: https://github.com/facebook/rocksdb/pull/10639 Reviewed By: cbi42 Differential Revision: D39263804 Pulled By: ajkr fbshipit-source-id: 9b5565bbafcb21a2f5b487c8364808dea2f0bc0c
2022-09-01Increase CircleCI no_output_timeout for macos-java builds (#10627)Peter Dillinger
Summary: ... because we are frequently seeing the 10m "no output" timeouts on these Pull Request resolved: https://github.com/facebook/rocksdb/pull/10627 Test Plan: CI Reviewed By: hx235 Differential Revision: D39224922 Pulled By: pdillinger fbshipit-source-id: f54c7adb5de87b2f57ccbc7f4e6c541b9cd37e08
2022-08-25CI benchmarks refine configuration (#10514)Alan Paxton
Summary: CI benchmarks refine configuration Run only “essential” benchmarks, but for longer Fix (reduce) the NUM_KEYS to ensure cached behaviour Reduce level size to try to ensure more levels Refine test durations again, more time per test, but fewer tests. In CI benchmark mode, the only read test is readrandom. There are still 3 mostly-read tests. Goal is to squeeze complete run a little bit inside 1 hour so it doesn’t clash with the next run (cron scheduled for main branch), but it gets to run as long as possible, so that results are as credible as possible. Reduce thread count to physical capacity, in an attempt to reduce throughput variance for write heavy tests. See Mark Callaghan’s comments in related documentation.. Pull Request resolved: https://github.com/facebook/rocksdb/pull/10514 Reviewed By: ajkr Differential Revision: D38952469 Pulled By: jay-zhuang fbshipit-source-id: 72fa6bba897cc47066ced65facd1fd36e28f30a8
2022-08-10Migrate to docker for CI run (#10496)Jay Zhuang
Summary: Moved linux builds to using docker to avoid CI instability caused by dependency installation site down. Added the `Dockerfile` which is used to build the image. The build time is also significantly reduced, because no dependencies installation and with using 2xlarge+ instance for slow build (like tsan test). Also fixed a few issues detected while building this: * `DestoryDB()` Status not checked for a few tests * nullptr might be used in `inlineskiplist.cc` Pull Request resolved: https://github.com/facebook/rocksdb/pull/10496 Test Plan: CI Reviewed By: ajkr Differential Revision: D38554200 Pulled By: jay-zhuang fbshipit-source-id: 16e8fb2bf07b9c84bb27fb18421c4d54f2f248fd
2022-08-08Tiered storage stress test (#10493)Jay Zhuang
Summary: Add Tiered storage stress test and db_bench option Pull Request resolved: https://github.com/facebook/rocksdb/pull/10493 Test Plan: new crashtest: https://app.circleci.com/pipelines/github/facebook/rocksdb/16905/workflows/68c2967c-9274-434f-8506-1403cf441ead Reviewed By: ajkr Differential Revision: D38481892 Pulled By: jay-zhuang fbshipit-source-id: 217a0be4acb93d420222e6ede2a1290d9f464776
2022-07-25Run new benchmark script in branch. (#10303)Alan Paxton
Summary: Configure CI to run modernised benchmark script Pull Request resolved: https://github.com/facebook/rocksdb/pull/10303 Reviewed By: ramvadiv Differential Revision: D37719116 Pulled By: jay-zhuang fbshipit-source-id: 79ecb1cd0abd4d800c6906ba6673268c2adee10e
2022-06-22Adapt benchmark result script to new fields. (#10120)Alan Paxton
Summary: Recently merged CI benchmark scripts were failing. There has clearly been a major revision of the fields of benchmark output. The upload script expects and sanity-checks the existence of some fields (changes date to conform to OpenSearch format)..., so the script needs to change. Also add a bit more exception checking to make it more obvious when this happens again. We have deleted the existing report.tsv from the benchmark machine. An existing report.tsv is appended to by default, so that if the fields change, later rows no longer match the header. This makes for an upload that dies half way through the report file, when the format no longer matches the header. Re-instate the config.yml for running the benchmarks, so we can once again test it in situ. Pull Request resolved: https://github.com/facebook/rocksdb/pull/10120 Reviewed By: pdillinger Differential Revision: D37314908 Pulled By: jay-zhuang fbshipit-source-id: 34f5243fee694b75c6838eb55d3398e4273254b2
2022-06-17Use optimized folly DistributedMutex in LRUCache when available (#10179)Peter Dillinger
Summary: folly DistributedMutex is faster than standard mutexes though imposes some static obligations on usage. See https://github.com/facebook/folly/blob/main/folly/synchronization/DistributedMutex.h for details. Here we use this alternative for our Cache implementations (especially LRUCache) for better locking performance, when RocksDB is compiled with folly. Also added information about which distributed mutex implementation is being used to cache_bench output and to DB LOG. Intended follow-up: * Use DMutex in more places, perhaps improving API to support non-scoped locking * Fix linking with fbcode compiler (needs ROCKSDB_NO_FBCODE=1 currently) Credit: Thanks Siying for reminding me about this line of work that was previously left unfinished. Pull Request resolved: https://github.com/facebook/rocksdb/pull/10179 Test Plan: for correctness, existing tests. CircleCI config updated. Also Meta-internal buck build updated. For performance, ran simultaneous before & after cache_bench. Out of three comparison runs, the middle improvement to ops/sec was +21%: Baseline: USE_CLANG=1 DEBUG_LEVEL=0 make -j24 cache_bench (fbcode compiler) ``` Complete in 20.201 s; Rough parallel ops/sec = 1584062 Thread ops/sec = 107176 Operation latency (ns): Count: 32000000 Average: 9257.9421 StdDev: 122412.04 Min: 134 Median: 3623.0493 Max: 56918500 Percentiles: P50: 3623.05 P75: 10288.02 P99: 30219.35 P99.9: 683522.04 P99.99: 7302791.63 ``` New: (add USE_FOLLY=1) ``` Complete in 16.674 s; Rough parallel ops/sec = 1919135 (+21%) Thread ops/sec = 135487 Operation latency (ns): Count: 32000000 Average: 7304.9294 StdDev: 108530.28 Min: 132 Median: 3777.6012 Max: 91030902 Percentiles: P50: 3777.60 P75: 10169.89 P99: 24504.51 P99.9: 59721.59 P99.99: 1861151.83 ``` Reviewed By: anand1976 Differential Revision: D37182983 Pulled By: pdillinger fbshipit-source-id: a17eb05f25b832b6a2c1356f5c657e831a5af8d1
2022-06-05Disable CI benchmark from #9723 (#10119)Jay Zhuang
Summary: The script is broken. Pull Request resolved: https://github.com/facebook/rocksdb/pull/10119 Reviewed By: ltamasi Differential Revision: D36928076 Pulled By: jay-zhuang fbshipit-source-id: f325cfd00869c506c64573fe8192cb5b561825d6
2022-06-04CI Benchmarking with CircleCI Runner and OpenSearch Dashboard (EB 1088) (#9723)Alan Paxton
Summary: CircleCI runner based benchmarking. A runner is a dedicate machine configured for CircleCI to perform work on. Our work is a repeatable benchmark, the `benchmark-linux` job in `config.yml` A runner, in CircleCI terminology, is a machine that is managed by the client (us) rather than running on CircleCI resources in the cloud. This means that we define and configure the iron, and that therefore the performance is repeatable and predictable. Which is what we need for performance regression benchmarking. On a time schedule (or on commit, during branch development) benchmarks are set off on the runner, and then a script is run `benchmark_log_tool.py` which parses the benchmark output and pushes it into a pre-configured OpenSearch document connected to an OpenSearch dashboard. Members of the team can examine benchmark performance changes on the dashboard. As time progresses we can add different benchmarks to the suite which gets run. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9723 Reviewed By: pdillinger Differential Revision: D35555626 Pulled By: jay-zhuang fbshipit-source-id: c6a905ca04494495c3784cfbb991f5ab90c807ee
2022-06-02Install zstd on CircleCI linux (#10102)Yanqin Jin
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/10102 Reviewed By: jay-zhuang Differential Revision: D36885468 Pulled By: riversand963 fbshipit-source-id: 6ed5b62dda8fe0f4be4b66d09bdec0134cf4500c
2022-05-20Point libprotobuf-mutator to the latest verified commit hash (#10028)Yanqin Jin
Summary: Recent updates to https://github.com/google/libprotobuf-mutator has caused link errors for RocksDB CircleCI job 'build-fuzzers'. This PR points the CI to a specific, most recent verified commit hash. Pull Request resolved: https://github.com/facebook/rocksdb/pull/10028 Test Plan: watch for CI to finish. Reviewed By: pdillinger, jay-zhuang Differential Revision: D36562517 Pulled By: riversand963 fbshipit-source-id: ba5ef0f9ed6ea6a75aa5dd2768bd5f389ac14f46
2022-05-11Remove slack CircleCI hook (#9982)Peter Dillinger
Summary: Our Slack site is deprecated Pull Request resolved: https://github.com/facebook/rocksdb/pull/9982 Test Plan: CircleCI Reviewed By: siying Differential Revision: D36322050 Pulled By: pdillinger fbshipit-source-id: 678202404d307e1547e4203d7e6bd467803ccd5e
2022-05-11Reorganize CircleCI workflows (#9981)Peter Dillinger
Summary: Condense down to 8 groups rather than 20+ for ease of browsing pages like https://app.circleci.com/pipelines/github/facebook/rocksdb?branch=main&filter=all Also, run nightly builds at 1AM or 2AM Pacific (depending on daylight time) rather than 4PM or 5PM Pacific, so that they actually use each day's landed changes. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9981 Test Plan: CI And manually inspected ``` grep -Eo 'build-[^: ]*' .circleci/config.yml | sort | uniq -c | less ``` to ensure I didn't orphan anything Reviewed By: jay-zhuang Differential Revision: D36317634 Pulled By: pdillinger fbshipit-source-id: 1c10d29d6b5d60ce3dd1364cd91f175380075ff3
2022-05-10Avoid some warnings-as-error in CircleCI+unity+AVX512F (#9978)Peter Dillinger
Summary: Example failure when compiling on sufficiently new hardware and built-in headers: ``` In file included from /usr/local/lib/gcc/x86_64-linux-gnu/12.1.0/include/immintrin.h:49, from ./util/bloom_impl.h:21, from table/block_based/filter_policy.cc:31, from unity.cc:167: In function '__m512i _mm512_shuffle_epi32(__m512i, _MM_PERM_ENUM)', inlined from 'void XXH3_accumulate_512_avx512(void*, const void*, const void*)' at util/xxhash.h:3605:58, inlined from 'void XXH3_accumulate(xxh_u64*, const xxh_u8*, const xxh_u8*, size_t, XXH3_f_accumulate_512)' at util/xxhash.h:4229:17, inlined from 'void XXH3_hashLong_internal_loop(xxh_u64*, const xxh_u8*, size_t, const xxh_u8*, size_t, XXH3_f_accumulate_512, XXH3_f_scrambleAcc)' at util/xxhash.h:4251:24, inlined from 'XXH128_hash_t XXH3_hashLong_128b_internal(const void*, size_t, const xxh_u8*, size_t, XXH3_f_accumulate_512, XXH3_f_scrambleAcc)' at util/xxhash.h:5065:32, inlined from 'XXH128_hash_t XXH3_hashLong_128b_withSecret(const void*, size_t, XXH64_hash_t, const void*, size_t)' at util/xxhash.h:5104:39: /usr/local/lib/gcc/x86_64-linux-gnu/12.1.0/include/avx512fintrin.h:4459:50: error: '__Y' may be used uninitialized [-Werror=maybe-uninitialized] ``` https://app.circleci.com/pipelines/github/facebook/rocksdb/13295/workflows/1695fb5c-40c1-423b-96b4-45107dc3012d/jobs/360416 Pull Request resolved: https://github.com/facebook/rocksdb/pull/9978 Test Plan: I was able to re-run in CircleCI with ssh, see the failure, ssh in and verify that adding -fno-avx512f fixed the failure. Will watch build-linux-unity-and-headers Reviewed By: riversand963 Differential Revision: D36296028 Pulled By: pdillinger fbshipit-source-id: ba5955cf2ac730f57d1d18c2f517e92f34be77a3
2022-05-10Increase soft open file limit for mini-crashtest on Linux (#9972)Peter Dillinger
Summary: CircleCI was using a soft open file limit of 1024 which would frequently be exceeded during test runs. Now using ``` ulimit -S -n `ulimit -H -n` ``` to set soft limit up to the hard limit (524288 in my test). I've also applied this same idiom to existing applicable MacOS configurations to reduce hard-coding numbers. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9972 Test Plan: CI Reviewed By: riversand963 Differential Revision: D36262943 Pulled By: pdillinger fbshipit-source-id: 86320cdf9b68a97fdb73531a7b4a59b4c2d2f73f
2022-05-09Fix format_compatible blowing away its TEST_TMPDIR (#9970)Peter Dillinger
Summary: https://github.com/facebook/rocksdb/issues/9961 broke format_compatible check because of `make clean` referencing TEST_TMPDIR. The Makefile behavior seems reasonable to me, so here's a fix in check_format_compatible.sh Apparently I also included removing a redundant part of our CircleCI config. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9970 Test Plan: manual run: SHORT_TEST=1 ./tools/check_format_compatible.sh Reviewed By: riversand963 Differential Revision: D36258172 Pulled By: pdillinger fbshipit-source-id: d46507f04614e888b414ff23b88d040ae2b5c294
2022-04-18Misc CI improvements / additions (#9859)Peter Dillinger
Summary: * Add valgrind test to nightly CircleCI (in case it can catch something that ASAN/UBSAN does not) * Add clang13+asan+ubsan+folly test to nightly CircleCI, for broader testing * Consolidate many copies of ASAN_OPTIONS= while also allowing it to be inherited from parent environment rather than always overridden. * Move UBSAN exclusion from Makefile into options_settable_test.cc Pull Request resolved: https://github.com/facebook/rocksdb/pull/9859 Test Plan: CI Reviewed By: jay-zhuang Differential Revision: D35730903 Pulled By: pdillinger fbshipit-source-id: 6f5464034e8115f9a07f6f7aec1de9219ec2837c
2022-04-18Add "no compression" job to CircleCI (#9850)Peter Dillinger
Summary: Since they operate at distinct abstraction layers, I thought it was prudent to combine with EncryptedEnv CI test for each PR, for efficiency in testing. Also added supported compressions to sst_dump --help output so that CI job can verify no compiled-in compression support. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9850 Test Plan: CI, some manual stuff Reviewed By: riversand963 Differential Revision: D35682346 Pulled By: pdillinger fbshipit-source-id: be9879c1533fed304ee32c89fd9ba4b07c2b90cc
2022-04-13Meta-internal folly integration with F14FastMap (#9546)Peter Dillinger
Summary: Especially after updating to C++17, I don't see a compelling case for *requiring* any folly components in RocksDB. I was able to purge the existing hard dependencies, and it can be quite difficult to strip out non-trivial components from folly for use in RocksDB. (The prospect of doing that on F14 has changed my mind on the best approach here.) But this change creates an optional integration where we can plug in components from folly at compile time, starting here with F14FastMap to replace std::unordered_map when possible (probably no public APIs for example). I have replaced the biggest CPU users of std::unordered_map with compile-time pluggable UnorderedMap which will use F14FastMap when USE_FOLLY is set. USE_FOLLY is always set in the Meta-internal buck build, and a simulation of that is in the Makefile for public CI testing. A full folly build is not needed, but checking out the full folly repo is much simpler for getting the dependency, and anything else we might want to optionally integrate in the future. Some picky details: * I don't think the distributed mutex stuff is actually used, so it was easy to remove. * I implemented an alternative to `folly::constexpr_log2` (which is much easier in C++17 than C++11) so that I could pull out the hard dependencies on `ConstexprMath.h` * I had to add noexcept move constructors/operators to some types to make F14's complainUnlessNothrowMoveAndDestroy check happy, and I added a macro to make that easier in some common cases. * Updated Meta-internal buck build to use folly F14Map (always) No updates to HISTORY.md nor INSTALL.md as this is not (yet?) considered a production integration for open source users. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9546 Test Plan: CircleCI tests updated so that a couple of them use folly. Most internal unit & stress/crash tests updated to use Meta-internal latest folly. (Note: they should probably use buck but they currently use Makefile.) Example performance improvement: when filter partitions are pinned in cache, they are tracked by PartitionedFilterBlockReader::filter_map_ and we can build a test that exercises that heavily. Build DB with ``` TEST_TMPDIR=/dev/shm/rocksdb ./db_bench -benchmarks=fillrandom -num=10000000 -disable_wal=1 -write_buffer_size=30000000 -bloom_bits=16 -compaction_style=2 -fifo_compaction_max_table_files_size_mb=10000 -fifo_compaction_allow_compaction=0 -partition_index_and_filters ``` and test with (simultaneous runs with & without folly, ~20 times each to see convergence) ``` TEST_TMPDIR=/dev/shm/rocksdb ./db_bench_folly -readonly -use_existing_db -benchmarks=readrandom -num=10000000 -bloom_bits=16 -compaction_style=2 -fifo_compaction_max_table_files_size_mb=10000 -fifo_compaction_allow_compaction=0 -partition_index_and_filters -duration=40 -pin_l0_filter_and_index_blocks_in_cache ``` Average ops/s no folly: 26229.2 Average ops/s with folly: 26853.3 (+2.4%) Reviewed By: ajkr Differential Revision: D34181736 Pulled By: pdillinger fbshipit-source-id: ffa6ad5104c2880321d8a1aa7187e00ab0d02e94
2022-03-30Upgrade gbenchmark to 1.6.1 (#9775)Jay Zhuang
Summary: Upgrade google benchmark to the latest 1.6.1. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9775 Test Plan: CI Reviewed By: riversand963 Differential Revision: D35252889 Pulled By: jay-zhuang fbshipit-source-id: 4d60dd1c6f522d0af0b3942ae8fa88e5ae17f34a
2022-03-24Fix make clean fail after java build (#9710)Jay Zhuang
Summary: Seems clean-rocksjava and clean-rocks conflict. Also remove unnecessary step in java CI build, otherwise it will rebuild the code again as java make sample do clean up first. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9710 Test Plan: `make rocksdbjava && make clean` should return success Reviewed By: riversand963 Differential Revision: D35122872 Pulled By: jay-zhuang fbshipit-source-id: 2a15b83e7a763c0fc0e42e1f35aac9551f951ece
2022-03-21Enable detect_stack_use_after_return for ASAN (#9714)Jay Zhuang
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/9714 Reviewed By: ajkr Differential Revision: D34983675 Pulled By: jay-zhuang fbshipit-source-id: 0252ec6ee38a0b960df4c92791c7c2bcbfba5ad8
generated by cgit v1.2.3-70-g09d2 (git 2.46.0) at 2024-10-22 22:27:34 +0000