[zfs] spl/zfs-0.6.5 released

#### New Feature Flags ####

* large_blocks - This feature allows the record size on a dataset
to be set larger than 128KB. We currently support block sizes from 512
bytes to 16MB. The benefits of larger blocks, and thus larger IO, need
to be weighed against the cost of COWing a giant block to modify one
byte. Additionally, very large blocks can have an impact on I/O
latency, and also potentially on the memory allocator. Therefore, we
do not allow the record size to be set larger than zfs_max_recordsize
(default 1MB). Larger blocks can be created by changing this tuning,
pools with larger blocks can always be imported and used, regardless of
this setting.

* filesystem_limits - This feature enables filesystem and snapshot
limits. These limits can be used to control how many filesystems and/or
snapshots can be created at the point in the tree on which the limits
are set.

#### Performance ####

* Improved zvol performance on all kernels
(>50% higher throughput, >20% lower latency)
* Improved zil performance on Linux 2.6.39 and earlier kernels
(10x lower latency)
* Improved allocation behavior on mostly full SSD/file pools
(5% to 10% improvement on 90% full pools)
* Improved performance when removing large files.
* Caching improvements (ARC):
* Better cached read performance due to reduced lock contention.
* Smarter heuristics for managing the total size of the cache and
the distribution of data/metadata.
* Faster release of cached buffers due to unexpected memory pressure.

#### Changes in Behavior ####

* Default reserved space was increased from 1.6% to 3.3% of total pool
capacity. This default percentage can be controlled through the new
spa_slop_shift module option, setting it to 6 will restore the
previous percentage.
* Loading of the ZFS module stack is now handled by systemd or the sysv
init scripts. Invoking the zfs/zpool commands will not cause the
modules to be automatically loaded. The previous behavior can be
restored by setting the `ZFS_MODULE_LOADING=yes` environment variable
but this functionality will be removed in a future release.
* Unified SYSV and Gentoo OpenRC initialization scripts. The previous
functionality has been split in to zfs-import, zfs-mount,
zfs-share, and zfs-zed scripts. This allows for independent
control of the services and is consistent with the unit files
provided for a systemd based system. Complete details of the
functionality provided by the updated scripts can be found at:
https://github.com/zfsonlinux/zfs/commit/2a34db1.
* Task queues are now dynamic and worker threads will be created and
destroyed as needed. This allows the system to automatically tune
itself to ensure the optimal number of threads are used for the active
workload which can result in a performance improvement.
* Task queue thread priorities were correctly aligned with the default
Linux file system thread priorities. This allows ZFS to compete
fairly with other active Linux file systems when the system is under
heavy load.
* When compression=on the default compression algorithm will be lz4 as
long as the feature is enabled. Otherwise the default remains
lzjb. Similarly lz4 is now the preferred method for compressing meta
data when available.
* The use of mkdir/rmdir/mv in the .zfs/snapshot directory has been
disabled by default both locally and via NFS clients. The
zfs_admin_snapshot module option can be used to re-enable this
functionality.
* LBA weighting is automatically disabled on files and SSDs ensuring
the entire device is used fairly.
* `iostat` accounting on zvols running on kernels older than Linux 3.19
is no longer supported.
* The known issues preventing swap on zvols for Linux 3.9 and newer
kernels have been resolved. However, deadlocks are still possible for
older kernels.

#### Module Options ####

* Changed zfs_arc_c_min default from 4M to 32M to accommodate large
blocks.
* Added metaslab_aliquot to control how many bytes are written to a
top-level vdev before moving on to the next one. Increasing this may
be helpful when using blocks larger than 1M.
* Added spa_slop_shift, see 'reserved space' comment in the 'changes to
behavior' section.
* Added zfs_admin_snapshot, enable/disable the use of mkdir/rmdir/mv in
.zfs/snapshot directory.
* Added zfs_arc_lotsfree_percent, throttle I/O when free system memory
drops below this percentage.
* Added zfs_arc_num_sublists_per_state, used to allow more fine-grained
locking.
* Added zfs_arc_p_min_shift, used to set a floor on arc_p.
* Added zfs_arc_sys_free, the target number of bytes the ARC should
leave as free.
* Added zfs_dbgmsg_enable, used to enable the 'dbgmsg' kstat.
* Added zfs_dbgmsg_maxsize, sets the maximum size of the dbgmsg buffer.
* Added zfs_max_recordsize, used to control the maximum allowed record
size.
* Added zfs_arc_meta_strategy, used to select the preferred ARC reclaim
strategy.
* Removed metaslab_min_alloc_size, it was unused internally due to
prior changes.
* Removed zfs_arc_memory_throttle_disable, replaced by
zfs_arc_lotsfree_percent.
* Removed zvol_threads, zvols no longer require a dedicated task queue.
* See `zfs-module-parameters(5)` for complete details on available
module options.

#### Bug Fixes ####

* Improved documentation with many updates, corrections, and additions.
* Improved sysv, systemd, initramfs, and dracut support.
* Improved block pointer validation before issuing IO.
* Improved scrub pause heuristics.
* Improved test coverage.
* Improved heuristics for automatic repair when `zfs_recover=1` module
option is set.
* Improved debugging infrastructure via 'dbgmsg' kstat.
* Improved `zpool import` performance.
* Fixed deadlocks in direct memory reclaim.
* Fixed deadlock on db_mtx and dn_holds.
* Fixed deadlock in dmu_objset_find_dp().
* Fixed deadlock during `zfs rollback`.
* Fixed kernel panic due to tsd_exit() in ZFS_EXIT.
* Fixed kernel panic when adding a duplicate dbuf to dn_dbufs.
* Fixed kernel panic due to security / ACL creation failure.
* Fixed kernel panic on unmount due to iput taskq.
* Fixed panic due to corrupt nvlist when running utilities.
* Fixed panic on unmount due to not waiting for all znodes to be
released.
* Fixed panic with `zfs clone` from different source and target pools.
* Fixed NULL pointer dereference in dsl_prop_get_ds().
* Fixed NULL pointer dereference in dsl_prop_notify_all_cb().
* Fixed NULL pointer dereference in zfsdev_getminor().
* Fixed I/Os are now aggregated across ZIO priority classes.
* Fixed .zfs/snapshot auto-mounting for all supported kernels.
* Fixed 3-digit octal escapes by changing to 4-digit which disambiguate
the output.
* Fixed hard lockup due to infinite loop in zfs_zget().
* Fixed misreported 'alloc' value for cache devices.
* Fixed spurious hung task watchdog stack traces.
* Fixed direct memory reclaim deadlocks.
* Fixed module loading in zfs import systemd service.
* Fixed intermittent libzfs_init() failure to open /dev/zfs.
* Fixed hot-disk sparing for disk vdevs
* Fixed system spinning during ARC reclaim.
* Fixed formatting errors in `zfs(8)`
* Fixed zio pipeline stall by having callers invoke next stage.
* Fixed assertion failed in zrl_tryenter().
* Fixed memory leak in make_root_vdev().
* Fixed memory leak in zpool_in_use().
* Fixed memory leak in libzfs when doing rollback.
* Fixed hold leak in dmu_recv_end_check().
* Fixed refcount leak in bpobj_iterate_impl().
* Fixed misuse of input argument in traverse_visitbp().
* Fixed missing missing mutex_destroy() calls.
* Fixed integer overflows in dmu_read/dmu_write.
* Fixed verify() failure in zio_done().
* Fixed zio_checksum_error() to only include info for ECKSUM errors.
* Fixed -ESTALE to force lookup on missing NFS file handles.
* Fixed spurious failures from dsl_dataset_hold_obj().
* Fixed zfs {ref}compressratio when using with 4k sector size.
* Fixed spurious watchdog warnings in prefetch thread.
* Fixed unfair disk space allocation when vdevs are of unequal size.
* Fixed ashift accounting error writing to cache devices.
* Fixed `zdb -d` has false positives when feature@large_blocks=disabled.
* Fixed `zdb -h | -i` seg fault.
* Fixed force-received full stream into a dataset if it has a snapshot.
* Fixed snapshot error handling.
* Fixed 'hangs' while deleting large files.
* Fixed lock contention (rrw_exit) while running a read only load.
* Fixed error message when creating pool include all problematic devs.
* Fixed Xen virtual block device detection, partitions are now created.
* Fixed missing E2BIG error handling in zfs_setprop_error().
* Fixed `zpool import` assertion in libzfs_import.c.
* Fixed `zfs send -nv` output to stderr.
* Fixed idle pool potentially running itself out of space.
* Fixed narrow race which allowed `read(2)` to access beyond
`fstat(2)`'s reported end-of-file.
* Fixed support for VPATH builds.
* Fixed double counting of HDR_L2ONLY_SIZE in ARC.
* Fixed 'BUG: Bad page state' warning from kernel due to writeback flag.
* Fixed arc_available_memory() to check freemem.
* Fixed arc_memory_throttle() to check pageout.
* Fixed`'zpool create` warning when using zvols in debug builds.
* Fixed loop devices layered on ZFS with 4.1 kernels.
* Fixed zvol contribution to kernel entropy pool.
* Fixed handling of compression flags in arc header.
* Substantial changes to realign code base with illumos.
* Many additional bug fixes.

==== SPL changes ====

Brian Behlendorf (17):
Fix cstyle issues in spl-tsd.c
Add boot_ncpus macro
Retire rwsem_is_locked() compat
Rename cv_wait_interruptible() to cv_wait_sig()
Add IMPLY() and EQUIV() macros
Add TASKQ_DYNAMIC feature
Set TASKQ_DYNAMIC for kmem and system taskqs
Add memory compatibility wrappers
Default to --disable-debug-kmem
Add defclsyspri macro
Remove skc_ref from alloc/free paths
Invert minclsyspri and maxclsyspri
Remove needfree, desfree, lotsfree #defines
Linux 4.2 compat: vfs_rename()
Revert "Create a new thread during recursive taskq dispatch if
necessary"
Linux 4.2 compat: misc_deregister()
Tag spl-0.6.5

Chris Dunlop (1):
Make taskq_wait() block until the queue is empty

Chunwei Chen (2):
Make tsd_set(key, NULL) remove the tsd entry for current thread
Restructure uio to accommodate bio_vec

Richard Yao (3):
vn_getf/vn_releasef should not accept negative file descriptors
Implement areleasef()
Disable direct reclaim in taskq worker threads on Linux 3.9+

Tim Chase (3):
Include other sources of freeable memory in the freemem calculation
Create a new thread during recursive taskq dispatch if necessary
Create a new thread during recursive taskq dispatch if necessary

Turbo Fredriksson (1):
Support parallel build trees (VPATH builds)

==== ZFS changes ====
Alek Pinchuk (1):
Illumos 6033 - arc_adjust() should search MFU lists

Alex Reece (4):
Illumos 3897 - zfs filesystem and snapshot limits (fix leak)
Illumos 4873 - zvol unmap calls can take a very long time for
larger datas
Illumos 5095 - panic when adding a duplicate dbuf to dn_dbufs
Illumos 5422 - preserve AVL invariants in dn_dbufs

Alexander Eremin (2):
Illumos 5847 - libzfs_diff should check zfs_prop_get() return
Illumos 5610 - zfs clone from different source and target pools
produces c

Alexander Motin (1):
Illumos 5008 - lock contention (rrw_exit) while running a read
only load

Andreas Buschmann (1):
Add extra keyword 'slot' to vdev_id.conf

Andrey Vesnovaty (2):
zfsctl: No need to sync ctldir inodes
Fix invalid fileid for snapshot root dentry

Andriy Gapon (5):
Account for ashift when gathering buffers to be written to l2arc
device
Illumos 5912 - full stream can not be force-received into a
dataset if it
Illumos 5909 - ensure that shared snap names don't become too long
after p
Illumos 5870 - dmu_recv_end_check() leaks origin_head hold if
error happen
Illumos 5946, 5945

Antonio Russo (1):
Relax restriction on zfs_ioc_next_obj() iteration

Arne Jansen (3):
Illumos 5269 - zpool import slow
Illumos 5981 - Deadlock in dmu_objset_find_dp
Illumos 6214 - zpools going south

Basil Crow (1):
Illumos 5118 - When verifying or creating a storage pool, error
messages o

Boris Protopopov (1):
Translate sync zio to sync bio

Brian Behlendorf (55):
Mark additional functions as PF_FSTRANS
Extend PF_FSTRANS critical regions
Merge branch 'zed-pushbullet'
Remove metaslab_min_alloc_size module option
Fix type mismatch on 32-bit systems
ztest should randomly change recordsize
Remove unused 'dsl_pool_t *dp' variable
Wait in libzfs_init() for the /dev/zfs device
Add libzfs_error_init() function
Use ExecStartPre to load zfs modules
Remove unused variable in vdev_add_child()
Add taskq_wait_outstanding() function
Use taskq_wait_outstanding() function
Make arc_prune() asynchronous
Merge branch 'lock-contention-on-arcs_mtx-final'
Rename cv_wait_interruptible() to cv_wait_sig()
Wait interruptibly in prefetch thread
Unify mount and share for 'zfs create/clone'
Increase the number of iput taskq threads
Add zfs_sb_prune_aliases() function
Add IMPLY() and EQUIV() macros
Update all default taskq settings
Add ziltest.sh
Move dracut directory to contrib
Use truncate instead of fallocate in ziltest.sh
Disable gcc bool-compare warning
Fix switch-bool warning
Linux 4.2 compat: bdi_setup_and_register()
Linux 4.2 compat: remove bio->bi_cnt access
Linux 4.2 compat: follow_link() / put_link()
Update inode under range lock
Add logic to try and recover an inode with an invalid mode
Set default _initconfdir directory
Reinstate zfs_arc_p_min_shift
Add hdr_recl() reclaim callback
Remove double counting HDR_L2ONLY_SIZE
Minor style cleanup
Check for NULL in dmu_free_long_range_impl()
Align thread priority with Linux defaults
Bound zvol_threads module option
Update arc_available_memory() to check freemem
Update arc_memory_throttle() to check pageout
Handle zap_lookup() failure in ddt_object_load()
Linux 4.2 compat: vfs_rename()
Limit max_hw_sectors_kb to 16M
Check large block feature flag on volumes
Linux 3.18 compat: Snapshot auto-mounting
Linux 4.2 compat: misc_deregister()
Add spa_slop_shift module option
Add temporary mount options
Merge branch 'zvol'
Support accessing .zfs/snapshot via NFS
Add dbgmsg kstat
Prefetch start and end of volumes
Tag zfs-0.6.5

Chris Dunlap (9):
Fix io-spare.sh to work with disk vdevs
Cleanup ZEDLETs
Replace "email" ZEDLETs with "notify" ZEDLETs
Add support for Pushbullet notifications
Add notification to io-spare.sh
Combine data-notify.sh with io-notify.sh
Update ZED copyright boilerplate
Fix whitespace in zed_log_err
Rework zed_notify_email for configurable PROG/OPTS

Chris Dunlop (4):
Wait for all znodes to be released before tearing down the superblock
arc_evict, arc_evict_ghost: reduce stack usage using kmem_zalloc
Linux 4.1 compat: configure bdi_setup_and_register()
Fix build failure with Linux 4.1 and FTRACE

Chris Williamson (1):
Illumos 5408 - managing ZFS cache devices requires lots of RAM

Christopher Siden (1):
Illumos 4951 - ZFS administrative commands (fix)

Chunwei Chen (4):
Fix kernel panic due to tsd_exit in ZFS_EXIT(zsb)
Fix "BUG: Bad page state" caused by writeback flag
Add compatibility layer for {kmap,kunmap}_atomic
Linux 4.1 compat: loop device on ZFS

Colin Ian King (1):
man: fix spelling mistakes in manual

DHE (2):
Rebuild init scripts on source file updates
Fix formatting error in zfs(8)

David Lamparter (1):
Safely handle security / ACL failures

Don Brady (2):
zdb -d has false positive warning when feature@large_blocks=disabled
Fix for recent zdb -h | -i crashes (seg fault)

Etienne Dechamps (3):
Allocate disk space fairly in the presence of vdevs of unequal size.
Document metaslab_aliquot.
Make metaslab_aliquot a module parameter.

Frédéric VANNIÈRE (1):
Fix build failure with Linux 4.1 and FTRACE

George Wilson (4):
Illumos #5244 - zio pipeline callers should explicitly invoke next
stage
Illumos 5369 - arc flags should be an enum
Illumos 5163 - arc should reap range_seg_cache
Illumos 4966 - zpool list iterator does not update output

Hajo Möller (2):
Change 3-digit octal escapes to 4-digit ones
Add -y option to `zpool iostat`

Isaac Huang (2):
Memory leak in make_root_vdev()
Remove useless variable spa_active_count

James Lee (1):
Reorder zfs-* services to allow /var on separate dataset

Jan Kryl (1):
Illumos 5427 - memory leak in libzfs when doing rollback

Jan Sanislo (1):
Return -ESTALE to force lookup for missing NFS file handles

Jason Zaman (1):
dmu: fix integer overflows

Jerry Jelinek (2):
Illumos 3897 - zfs filesystem and snapshot limits
Illumos 4901 - zfs filesystem/snapshot limit leaks

Jorgen Lundman (1):
Illumos 4975 - missing mutex_destroy() calls in zfs

Josef 'Jeff' Sipek (2):
Illumos 4626 - libzfs memleak in zpool_in_use()
Illumos 4745 - fix AVL code misspellings

Justin T. Gibbs (10):
5313 Allow I/Os to be aggregated across ZIO priority classes
Illumos 5038 - Remove "old-style" flexible array usage in ZFS.
Illumos 5310 - Remove always true tests for non-NULL ds->ds_phys
Illumos 5314 - Remove "dbuf phys" db->db_data pointer aliases in ZFS
Illumos 5056 - ZFS deadlock on db_mtx and dn_holds
Illumos 5531 - NULL pointer dereference in dsl_prop_get_ds()
Illumos 5592 - NULL pointer dereference in dsl_prop_notify_all_cb()
Illumos 5562 - ZFS sa_handle's violate kmem invariants, debug
kernels pani
Illumos 5393 - spurious failures from dsl_dataset_hold_obj()
Illumos 5661 - ZFS: "compression = on" should use lz4 if feature
is enable

Manoj Joseph (1):
Illumos 5764 - "zfs send -nv" directs output to stderr

Matthew Ahrens (19):
Illumos 5134 - if ZFS_DEBUG or debug= is set, libzpool should
enable debug
Illumos 5812 - assertion failed in zrl_tryenter(): zr_owner==NULL
Illumos 3654,3656
Illumos 4951 - ZFS administrative commands should use reserved space
Illumos 5808 - spa_check_logs is not necessary on readonly pools
Illumos 5820 - verify failed in zio_done(): BP_EQUAL(bp, io_bp_orig)
Illumos 5348 - zio_checksum_error() only fills in info if ECKSUM
Illumos 5349 - verify that block pointer is plausible before reading
Illumos 5027 - zfs large block support
Illumos 5350 - clean up code in dnode_sync()
Illumos 5351, 5352 - scrub pauses
Illumos 5810 - zdb should print details of bpobj
Illumos 5243 - zdb -b could be much faster
Illumos 5818 - zfs {ref}compressratio is incorrect with 4k sector size
Illumos 5368 - ARC should cache more metadata
Illumos 5175 - implement dmu_read_uio_dbuf() to improve cached
read perfor
Illumos 5911 - ZFS "hangs" while deleting file
Illumos 5347 - idle pool may run itself out of space
Illumos 5376 - arc_kmem_reap_now() should not result in clearing
arc_no_gr

Matus Kral (1):
Linux 4.1 compat: use read_iter() / write_iter()

Max Grossman (2):
Illumos 4373 - add block contents print to zstreamdump
Illumos 5765 - add support for estimating send stream size with
lzc_send_s

Ned Bass (3):
Serialize access to spa->spa_feat_stats nvlist
dbuf_try_add_ref minor bug fixes
dmu_objset_userquota_get_ids uses dn_bonus unsafely

Paul B. Henson (1):
5410 Document -S option to zfs inherit

Prakash Surya (4):
Illumos 5497 - lock contention on arcs_mtx
Illumos 5701 - zpool list reports incorrect "alloc" value for
cache device
Illumos 5445 - Add more visibility via arcstats
Illumos 5817 - change type of arcs_size from uint64_t to refcount_t

Prasad Joshi (1):
Illumos 1765 - assert triggered in libzfs_import.c

Richard Yao (22):
zfsdev_getminor() should check for invalid file handles
Fix Xen Virtual Block Device detection
Failure of userland copy should return EFAULT
Fix zvol detection
Some OpenRC dependency logic belongs in mount
Clear QUEUE_FLAG_ADD_RANDOM on zvols
Remove blk_queue_physical_block_size() autotools check
Remove blk_queue_io_opt() autotools check
Disable LBA weighting on files and SSDs
VDEV_REQ_FUA should be mapped to REQ_FUA
zvol processing should use struct bio
Support secure discard on zvols
Remove rq_for_each_segment() autotools check
Remove rq_is_sync() autotools check
Remove blk_end_request() autotools check.
Remove blk_requeue_request() autotools check
Remove blk_fetch_request() autotools check
Remove blk_rq_pos() autotools check
Remove blk_rq_bytes()/blk_rq_sectors autotools checks
Remove blk_queue_discard() autotools check
Remove blk_queue_nonrot() autotools check
Reintroduce IO accounting on zvols on Linux 3.19+

SenH (1):
Force create /run/sendsigs.omit.d link when starting zed

Sören Tempel (4):
Refactor dracut module
Force export if it failed the first time
Only source '/lib/dracut-lib.sh' if it wasn't so far
Update dracut README

Tim Chase (18):
Allocate zfs_znode_cache on the Linux slab
Support the "version" property on volumes via the
zfs_prop_get_int() API
Linux 2.6.36 compat, use REQ_FAILFAST_MASK and remove pre-2.6.36
support
Zdb should be able to open the root dataset
Initialize dbu_tqent in dmu_buf_init_user()
Revert "arc_evict, arc_evict_ghost: reduce stack usage using
kmem_zalloc"
Revert "fix l2arc compression buffers leak"
Revert "Allow arc_evict_ghost() to only evict meta data"
Partially revert "Add ddt, ddt_entry, and l2arc_hdr caches"
Increase arc_c_min to allow safe operation of arc_adapt()
3.12 compat, NUMA-aware per-superblock shrinker
Remove l2ad_evict from zfs_l2arc_evict_class
Prevent reclaim in metaslab preload threads
ztest: display non-index properties properly at verbose level 6
Allow recovery from corrupted snapshot maps
Initialize the taskq entry embedded within struct vdev
Dbuf hash table should be sized as is the arc hash table
Prevent reclaim in the traverse prefetch thread

Turbo Fredriksson (15):
Document bookmarks a little better in zfs(8)
Add the '-a' option to 'zpool export'
The mount helper mount.zfs MUST be in /sbin (not '$sbindir').
Style check shell scripts
Base init scripts for SYSV systems
SYSV init script fixes.
Improve on the ZFS events documentation
Additional SYSV init script fixes.
Revert "Additional SYSV init script fixes."
Additional SYSV init script fixes (3).
Add /dev/mapper to the list of possible sources for pool devices.
Initramfs scripts for ZoL.
Support parallel build trees (VPATH builds)
Fix some minor issues with the SYSV init and initramfs scripts.
Make sure that POOL_IMPORTED is set, unset and checked where
appropriate.

Will Andrews (2):
Illumos 5814 - bpobj_iterate_impl(): Close a refcount leak
iterating on a
Illumos 5813 - zfs_setprop_error(): Handle errno value E2BIG.

cburroughs (1):
align zfs_autoimport_disable manpage with reality

loli10K (1):
Fix small typo

tuxoko (4):
Fix misuse of input argument in traverse_visitbp
Add cond_resched to zfs_zget to prevent infinite loop
Add parenthesis to the ternary operator
Check for RW_WRITE_HELD in zfs_inactive