Return the right error value when dup[23]() newfd argument is too large

Jack Lin reports that the error return from dup3() for the RLIMIT_NOFILE
case changed incorrectly after 3.6.

The culprit is commit f33ff9927f ("take rlimit check to callers of
expand_files()") which when it moved the "return -EMFILE" out to the
caller, didn't notice that the dup3() had special code to turn the
EMFILE return into EBADF.

The replace_fd() helper that got added later then inherited the bug too.

Reported-by: Jack Lin <linliangjie@huawei.com>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
[ Noted more bugs, wrote proper changelog, fixed up typos - Linus ]
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
This commit is contained in:
Al Viro 2012-10-31 03:37:48 +00:00 committed by Linus Torvalds
parent 2df4f26167
commit 08f05c4974

View file

@ -900,7 +900,7 @@ int replace_fd(unsigned fd, struct file *file, unsigned flags)
return __close_fd(files, fd);
if (fd >= rlimit(RLIMIT_NOFILE))
return -EMFILE;
return -EBADF;
spin_lock(&files->file_lock);
err = expand_files(files, fd);
@ -926,7 +926,7 @@ SYSCALL_DEFINE3(dup3, unsigned int, oldfd, unsigned int, newfd, int, flags)
return -EINVAL;
if (newfd >= rlimit(RLIMIT_NOFILE))
return -EMFILE;
return -EBADF;
spin_lock(&files->file_lock);
err = expand_files(files, newfd);