git fsck failure on OS X with files >= 4 GiB

classic Classic list List threaded Threaded
34 messages Options
12
Reply | Threaded
Open this post in threaded view
|

git fsck failure on OS X with files >= 4 GiB

Rafael Espíndola
I first noticed this with "2.4.9 (Apple Git-60)", but it reproduces
with git built from 37023ba381b6d251d7140a997b39b566dbc63c42.

Create two files with just 0s:

-rw-r--r--  1 espindola  staff  4294967296 28 Oct 11:09 exactly-4gib
-rw-r--r--  1 espindola  staff  4294967295 28 Oct 11:09 one-less-than-4gib


and run

git init
git add one-less-than-4gib
git commit -m bar
git fsck
git add exactly-4gib
git commit -m bar
git fsck

The first fsck will run with no problems, but the second one fails:

error: packed cfdaf54c9ccfd8f5e4cee562f7d5f92df13d3106 from
.git/objects/pack/pack-ff08480fd7f767b6bd0aeb559f0f5dea2245b0b3.pack
is corrupt

Using the very same revision on freebsd doesn't cause any errors.

Cheers,
Rafael
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

Re: git fsck failure on OS X with files >= 4 GiB

Filipe Cabecinhas
I did some debugging, and it seems CC_SHA1_Update (used by
write_sha1_file_prepare if APPLE_COMMON_CRYPTO is defined in the
Makefile) takes a uint32_t as a "length" parameter, which explains why
it stops working at 4GiB (UINT_MAX+1).

In the OS X 10.11 SDK header CommonCrypto/CommonDigest.h, we have:

typedef uint32_t CC_LONG;       /* 32 bit unsigned integer */
//...
extern int CC_SHA1_Update(CC_SHA1_CTX *c, const void *data, CC_LONG len)

A possible fix would be to either call SHA1_Update with a maximum of
UINT_MAX, looping if necessary. Or have a compatibility SHA1_Update
for OS X which can handle data longer than UINT_MAX.

I'm not sure what the git maintainers would prefer.

Regards,

  Filipe


On Wed, Oct 28, 2015 at 4:10 PM, Rafael Espíndola
<[hidden email]> wrote:

>
> I first noticed this with "2.4.9 (Apple Git-60)", but it reproduces
> with git built from 37023ba381b6d251d7140a997b39b566dbc63c42.
>
> Create two files with just 0s:
>
> -rw-r--r--  1 espindola  staff  4294967296 28 Oct 11:09 exactly-4gib
> -rw-r--r--  1 espindola  staff  4294967295 28 Oct 11:09 one-less-than-4gib
>
>
> and run
>
> git init
> git add one-less-than-4gib
> git commit -m bar
> git fsck
> git add exactly-4gib
> git commit -m bar
> git fsck
>
> The first fsck will run with no problems, but the second one fails:
>
> error: packed cfdaf54c9ccfd8f5e4cee562f7d5f92df13d3106 from
> .git/objects/pack/pack-ff08480fd7f767b6bd0aeb559f0f5dea2245b0b3.pack
> is corrupt
>
> Using the very same revision on freebsd doesn't cause any errors.
>
> Cheers,
> Rafael
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

Re: git fsck failure on OS X with files >= 4 GiB

Rafael Espíndola
In reply to this post by Rafael Espíndola
Awesome, building with

NO_OPENSSL = 1
NO_GETTEXT = 1

produces a working git :-)

Cheers,
Rafael


On 28 October 2015 at 23:37, Filipe Cabecinhas <[hidden email]> wrote:

> I did some debugging, and it seems CC_SHA1_Update (used by
> write_sha1_file_prepare if APPLE_COMMON_CRYPTO is defined in the Makefile)
> takes a uint32_t as a "length" parameter, which explains why it stops
> working at 4GiB (UINT_MAX+1).
>
> In the OS X 10.11 SDK header CommonCrypto/CommonDigest.h, we have:
>
> typedef uint32_t CC_LONG;       /* 32 bit unsigned integer */
> //...
> extern int CC_SHA1_Update(CC_SHA1_CTX *c, const void *data, CC_LONG len)
>
> A possible fix would be to either call SHA1_Update with a maximum of
> UINT_MAX, looping if necessary. Or have a compatibility SHA1_Update for OS X
> which can handle data longer than UINT_MAX.
>
> I'm not sure what the git maintainers would prefer.
>
> Regards,
>
>   Filipe
>
> On Wed, Oct 28, 2015 at 4:10 PM, Rafael Espíndola
> <[hidden email]> wrote:
>>
>> I first noticed this with "2.4.9 (Apple Git-60)", but it reproduces
>> with git built from 37023ba381b6d251d7140a997b39b566dbc63c42.
>>
>> Create two files with just 0s:
>>
>> -rw-r--r--  1 espindola  staff  4294967296 28 Oct 11:09 exactly-4gib
>> -rw-r--r--  1 espindola  staff  4294967295 28 Oct 11:09 one-less-than-4gib
>>
>>
>> and run
>>
>> git init
>> git add one-less-than-4gib
>> git commit -m bar
>> git fsck
>> git add exactly-4gib
>> git commit -m bar
>> git fsck
>>
>> The first fsck will run with no problems, but the second one fails:
>>
>> error: packed cfdaf54c9ccfd8f5e4cee562f7d5f92df13d3106 from
>> .git/objects/pack/pack-ff08480fd7f767b6bd0aeb559f0f5dea2245b0b3.pack
>> is corrupt
>>
>> Using the very same revision on freebsd doesn't cause any errors.
>>
>> Cheers,
>> Rafael
>
>
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

Re: git fsck failure on OS X with files >= 4 GiB

Filipe Cabecinhas
Defining BLK_SHA1 = YesPlease (when calling make) should just change
the SHA functions, instead of completely removing OpenSSL or
CommonCrypto.

Regards,
  Filipe


On Thu, Oct 29, 2015 at 3:46 AM, Rafael Espíndola
<[hidden email]> wrote:

> Awesome, building with
>
> NO_OPENSSL = 1
> NO_GETTEXT = 1
>
> produces a working git :-)
>
> Cheers,
> Rafael
>
>
> On 28 October 2015 at 23:37, Filipe Cabecinhas <[hidden email]> wrote:
>> I did some debugging, and it seems CC_SHA1_Update (used by
>> write_sha1_file_prepare if APPLE_COMMON_CRYPTO is defined in the Makefile)
>> takes a uint32_t as a "length" parameter, which explains why it stops
>> working at 4GiB (UINT_MAX+1).
>>
>> In the OS X 10.11 SDK header CommonCrypto/CommonDigest.h, we have:
>>
>> typedef uint32_t CC_LONG;       /* 32 bit unsigned integer */
>> //...
>> extern int CC_SHA1_Update(CC_SHA1_CTX *c, const void *data, CC_LONG len)
>>
>> A possible fix would be to either call SHA1_Update with a maximum of
>> UINT_MAX, looping if necessary. Or have a compatibility SHA1_Update for OS X
>> which can handle data longer than UINT_MAX.
>>
>> I'm not sure what the git maintainers would prefer.
>>
>> Regards,
>>
>>   Filipe
>>
>> On Wed, Oct 28, 2015 at 4:10 PM, Rafael Espíndola
>> <[hidden email]> wrote:
>>>
>>> I first noticed this with "2.4.9 (Apple Git-60)", but it reproduces
>>> with git built from 37023ba381b6d251d7140a997b39b566dbc63c42.
>>>
>>> Create two files with just 0s:
>>>
>>> -rw-r--r--  1 espindola  staff  4294967296 28 Oct 11:09 exactly-4gib
>>> -rw-r--r--  1 espindola  staff  4294967295 28 Oct 11:09 one-less-than-4gib
>>>
>>>
>>> and run
>>>
>>> git init
>>> git add one-less-than-4gib
>>> git commit -m bar
>>> git fsck
>>> git add exactly-4gib
>>> git commit -m bar
>>> git fsck
>>>
>>> The first fsck will run with no problems, but the second one fails:
>>>
>>> error: packed cfdaf54c9ccfd8f5e4cee562f7d5f92df13d3106 from
>>> .git/objects/pack/pack-ff08480fd7f767b6bd0aeb559f0f5dea2245b0b3.pack
>>> is corrupt
>>>
>>> Using the very same revision on freebsd doesn't cause any errors.
>>>
>>> Cheers,
>>> Rafael
>>
>>
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

Re: git fsck failure on OS X with files >= 4 GiB

atousa.p
Here is a solution that avoids problems with OS-specific
implementations of SHA_Update() by limiting the size of each update
request to 1GiB and calling the function repeatedly in a loop.

Atousa

--

[PATCH] Limit the size of the data block passed to SHA1_Update()

This avoids issues where OS-specific implementations use
a 32-bit integer to specify block size.  Limit currently
set to 1GiB.
---
 cache.h | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)

diff --git a/cache.h b/cache.h
index 79066e5..c305985 100644
--- a/cache.h
+++ b/cache.h
@@ -14,10 +14,28 @@
 #ifndef git_SHA_CTX
 #define git_SHA_CTX SHA_CTX
 #define git_SHA1_Init SHA1_Init
-#define git_SHA1_Update SHA1_Update
 #define git_SHA1_Final SHA1_Final
 #endif

+#define SHA1_MAX_BLOCK_SIZE (1024*1024*1024)
+
+static inline int git_SHA1_Update(SHA_CTX *c, const void *data, size_t len)
+{
+ size_t nr;
+ size_t total = 0;
+ char *cdata = (char*)data;
+ while(len > 0) {
+ nr = len;
+ if(nr > SHA1_MAX_BLOCK_SIZE)
+ nr = SHA1_MAX_BLOCK_SIZE;
+ SHA1_Update(c, cdata, nr);
+ total += nr;
+ cdata += nr;
+ len -= nr;
+ }
+ return total;
+}
+
 #include <zlib.h>
 typedef struct git_zstream {
  z_stream z;
--
2.4.9 (Apple Git-60)


On Thu, Oct 29, 2015 at 8:15 AM, Filipe Cabecinhas <[hidden email]> wrote:

> Defining BLK_SHA1 = YesPlease (when calling make) should just change
> the SHA functions, instead of completely removing OpenSSL or
> CommonCrypto.
>
> Regards,
>   Filipe
>
>
> On Thu, Oct 29, 2015 at 3:46 AM, Rafael Espíndola
> <[hidden email]> wrote:
>> Awesome, building with
>>
>> NO_OPENSSL = 1
>> NO_GETTEXT = 1
>>
>> produces a working git :-)
>>
>> Cheers,
>> Rafael
>>
>>
>> On 28 October 2015 at 23:37, Filipe Cabecinhas <[hidden email]> wrote:
>>> I did some debugging, and it seems CC_SHA1_Update (used by
>>> write_sha1_file_prepare if APPLE_COMMON_CRYPTO is defined in the Makefile)
>>> takes a uint32_t as a "length" parameter, which explains why it stops
>>> working at 4GiB (UINT_MAX+1).
>>>
>>> In the OS X 10.11 SDK header CommonCrypto/CommonDigest.h, we have:
>>>
>>> typedef uint32_t CC_LONG;       /* 32 bit unsigned integer */
>>> //...
>>> extern int CC_SHA1_Update(CC_SHA1_CTX *c, const void *data, CC_LONG len)
>>>
>>> A possible fix would be to either call SHA1_Update with a maximum of
>>> UINT_MAX, looping if necessary. Or have a compatibility SHA1_Update for OS X
>>> which can handle data longer than UINT_MAX.
>>>
>>> I'm not sure what the git maintainers would prefer.
>>>
>>> Regards,
>>>
>>>   Filipe
>>>
>>> On Wed, Oct 28, 2015 at 4:10 PM, Rafael Espíndola
>>> <[hidden email]> wrote:
>>>>
>>>> I first noticed this with "2.4.9 (Apple Git-60)", but it reproduces
>>>> with git built from 37023ba381b6d251d7140a997b39b566dbc63c42.
>>>>
>>>> Create two files with just 0s:
>>>>
>>>> -rw-r--r--  1 espindola  staff  4294967296 28 Oct 11:09 exactly-4gib
>>>> -rw-r--r--  1 espindola  staff  4294967295 28 Oct 11:09 one-less-than-4gib
>>>>
>>>>
>>>> and run
>>>>
>>>> git init
>>>> git add one-less-than-4gib
>>>> git commit -m bar
>>>> git fsck
>>>> git add exactly-4gib
>>>> git commit -m bar
>>>> git fsck
>>>>
>>>> The first fsck will run with no problems, but the second one fails:
>>>>
>>>> error: packed cfdaf54c9ccfd8f5e4cee562f7d5f92df13d3106 from
>>>> .git/objects/pack/pack-ff08480fd7f767b6bd0aeb559f0f5dea2245b0b3.pack
>>>> is corrupt
>>>>
>>>> Using the very same revision on freebsd doesn't cause any errors.
>>>>
>>>> Cheers,
>>>> Rafael
>>>
>>>
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to [hidden email]
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



--
Atousa Pahlevan, PhD
M.Math. University of Waterloo, Canada
Ph.D. Department of Computer Science, University of Victoria, Canada
Voice: 415-341-6206
Email: [hidden email]
Website: www.apahlevan.org
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

Re: git fsck failure on OS X with files >= 4 GiB

Junio C Hamano
Atousa Duprat <[hidden email]> writes:

> [PATCH] Limit the size of the data block passed to SHA1_Update()
>
> This avoids issues where OS-specific implementations use
> a 32-bit integer to specify block size.  Limit currently
> set to 1GiB.
> ---
>  cache.h | 20 +++++++++++++++++++-
>  1 file changed, 19 insertions(+), 1 deletion(-)
>
> diff --git a/cache.h b/cache.h
> index 79066e5..c305985 100644
> --- a/cache.h
> +++ b/cache.h
> @@ -14,10 +14,28 @@
>  #ifndef git_SHA_CTX
>  #define git_SHA_CTX SHA_CTX
>  #define git_SHA1_Init SHA1_Init
> -#define git_SHA1_Update SHA1_Update
>  #define git_SHA1_Final SHA1_Final
>  #endif
>
> +#define SHA1_MAX_BLOCK_SIZE (1024*1024*1024)
> +
> +static inline int git_SHA1_Update(SHA_CTX *c, const void *data, size_t len)
> +{
> + size_t nr;
> + size_t total = 0;
> + char *cdata = (char*)data;
> + while(len > 0) {
> + nr = len;
> + if(nr > SHA1_MAX_BLOCK_SIZE)
> + nr = SHA1_MAX_BLOCK_SIZE;
> + SHA1_Update(c, cdata, nr);
> + total += nr;
> + cdata += nr;
> + len -= nr;
> + }
> + return total;
> +}
> +

I think the idea illustrated above is a good start, but there are
a few issues:

 * SHA1_Update() is used in fairly many places; it is unclear if it
   is a good idea to inline.

 * There is no need to punish implementations with working
   SHA1_Update by another level of wrapping.

 * What would you do when you find an implementation for which 1G is
   still too big?

Perhaps something like this in the header

#ifdef SHA1_MAX_BLOCK_SIZE
extern int SHA1_Update_Chunked(SHA_CTX *, const void *, size_t);
#define git_SHA1_Update SHA1_Update_Chunked
#endif

with compat/sha1_chunked.c that has

#ifdef SHA1_MAX_BLOCK_SIZE
int SHA1_Update_Chunked(SHA_CTX *c, const void *data, size_t len)
{
        ... your looping implementation ...
}
#endif

in it, that is only triggered via a Makefile macro, e.g.
might be a good workaround.

diff --git a/Makefile b/Makefile
index 8466333..83348b8 100644
--- a/Makefile
+++ b/Makefile
@@ -139,6 +139,10 @@ all::
 # Define PPC_SHA1 environment variable when running make to make use of
 # a bundled SHA1 routine optimized for PowerPC.
 #
+# Define SHA1_MAX_BLOCK_SIZE if your SSH1_Update() implementation can
+# hash only a limited amount of data in one call (e.g. APPLE_COMMON_CRYPTO
+# may want 'SHA1_MAX_BLOCK_SIZE=1024L*1024L*1024L' defined).
+#
 # Define NEEDS_CRYPTO_WITH_SSL if you need -lcrypto when using -lssl (Darwin).
 #
 # Define NEEDS_SSL_WITH_CRYPTO if you need -lssl when using -lcrypto (Darwin).
@@ -1002,6 +1006,7 @@ ifeq ($(uname_S),Darwin)
  ifndef NO_APPLE_COMMON_CRYPTO
  APPLE_COMMON_CRYPTO = YesPlease
  COMPAT_CFLAGS += -DAPPLE_COMMON_CRYPTO
+ SHA1_MAX_BLOCK_SIZE=1024L*1024L*1024L
  endif
  NO_REGEX = YesPlease
  PTHREAD_LIBS =
@@ -1350,6 +1355,11 @@ endif
 endif
 endif
 
+ifdef SHA1_MAX_BLOCK_SIZE
+LIB_OBJS += compat/sha1_chunked.o
+BASIC_CFLAGS += SHA1_MAX_BLOCK_SIZE="$(SHA1_MAX_BLOCK_SIZE)"
+endif
+
 ifdef NO_PERL_MAKEMAKER
  export NO_PERL_MAKEMAKER
 endif
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

Re: git fsck failure on OS X with files >= 4 GiB

atousa.p
Thank you for the feedback.  I have revised the proposed patch as
suggested, allowing the use of SHA1_MAX_BLOCK_SIZE to enable the
chunked implementation.  When building for OSX with the CommonCrypto
library we error out if SHA1_MAX_BLOCK_SIZE is not defined, which will
avoid compiling a version of the tool that won't compute hashes
properly on large files.  It should be easy to enable this on other
platforms if needed.

Atousa

On Thu, Oct 29, 2015 at 10:19 AM, Junio C Hamano <[hidden email]> wrote:

> Atousa Duprat <[hidden email]> writes:
>
>> [PATCH] Limit the size of the data block passed to SHA1_Update()
>>
>> This avoids issues where OS-specific implementations use
>> a 32-bit integer to specify block size.  Limit currently
>> set to 1GiB.
>> ---
>>  cache.h | 20 +++++++++++++++++++-
>>  1 file changed, 19 insertions(+), 1 deletion(-)
>>
>> diff --git a/cache.h b/cache.h
>> index 79066e5..c305985 100644
>> --- a/cache.h
>> +++ b/cache.h
>> @@ -14,10 +14,28 @@
>>  #ifndef git_SHA_CTX
>>  #define git_SHA_CTX SHA_CTX
>>  #define git_SHA1_Init SHA1_Init
>> -#define git_SHA1_Update SHA1_Update
>>  #define git_SHA1_Final SHA1_Final
>>  #endif
>>
>> +#define SHA1_MAX_BLOCK_SIZE (1024*1024*1024)
>> +
>> +static inline int git_SHA1_Update(SHA_CTX *c, const void *data, size_t len)
>> +{
>> + size_t nr;
>> + size_t total = 0;
>> + char *cdata = (char*)data;
>> + while(len > 0) {
>> + nr = len;
>> + if(nr > SHA1_MAX_BLOCK_SIZE)
>> + nr = SHA1_MAX_BLOCK_SIZE;
>> + SHA1_Update(c, cdata, nr);
>> + total += nr;
>> + cdata += nr;
>> + len -= nr;
>> + }
>> + return total;
>> +}
>> +
>
> I think the idea illustrated above is a good start, but there are
> a few issues:
>
>  * SHA1_Update() is used in fairly many places; it is unclear if it
>    is a good idea to inline.
>
>  * There is no need to punish implementations with working
>    SHA1_Update by another level of wrapping.
>
>  * What would you do when you find an implementation for which 1G is
>    still too big?
>
> Perhaps something like this in the header
>
> #ifdef SHA1_MAX_BLOCK_SIZE
> extern int SHA1_Update_Chunked(SHA_CTX *, const void *, size_t);
> #define git_SHA1_Update SHA1_Update_Chunked
> #endif
>
> with compat/sha1_chunked.c that has
>
> #ifdef SHA1_MAX_BLOCK_SIZE
> int SHA1_Update_Chunked(SHA_CTX *c, const void *data, size_t len)
> {
>         ... your looping implementation ...
> }
> #endif
>
> in it, that is only triggered via a Makefile macro, e.g.
> might be a good workaround.
>
> diff --git a/Makefile b/Makefile
> index 8466333..83348b8 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -139,6 +139,10 @@ all::
>  # Define PPC_SHA1 environment variable when running make to make use of
>  # a bundled SHA1 routine optimized for PowerPC.
>  #
> +# Define SHA1_MAX_BLOCK_SIZE if your SSH1_Update() implementation can
> +# hash only a limited amount of data in one call (e.g. APPLE_COMMON_CRYPTO
> +# may want 'SHA1_MAX_BLOCK_SIZE=1024L*1024L*1024L' defined).
> +#
>  # Define NEEDS_CRYPTO_WITH_SSL if you need -lcrypto when using -lssl (Darwin).
>  #
>  # Define NEEDS_SSL_WITH_CRYPTO if you need -lssl when using -lcrypto (Darwin).
> @@ -1002,6 +1006,7 @@ ifeq ($(uname_S),Darwin)
>         ifndef NO_APPLE_COMMON_CRYPTO
>                 APPLE_COMMON_CRYPTO = YesPlease
>                 COMPAT_CFLAGS += -DAPPLE_COMMON_CRYPTO
> +               SHA1_MAX_BLOCK_SIZE=1024L*1024L*1024L
>         endif
>         NO_REGEX = YesPlease
>         PTHREAD_LIBS =
> @@ -1350,6 +1355,11 @@ endif
>  endif
>  endif
>
> +ifdef SHA1_MAX_BLOCK_SIZE
> +LIB_OBJS += compat/sha1_chunked.o
> +BASIC_CFLAGS += SHA1_MAX_BLOCK_SIZE="$(SHA1_MAX_BLOCK_SIZE)"
> +endif
> +
>  ifdef NO_PERL_MAKEMAKER
>         export NO_PERL_MAKEMAKER
>  endif



--
Atousa Pahlevan, PhD
M.Math. University of Waterloo, Canada
Ph.D. Department of Computer Science, University of Victoria, Canada
Voice: 415-341-6206
Email: [hidden email]
Website: www.apahlevan.org
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

[PATCH] Limit the size of the data block passed to SHA1_Update()

atousa.p
Some implementations of SHA_Updates have inherent limits
on the max chunk size. SHA1_MAX_BLOCK_SIZE can be defined
to set the max chunk size supported, if required.  This is
enabled for OSX CommonCrypto library and set to 1GiB.
---
 Makefile                     |  9 +++++++++
 cache.h                      |  7 ++++++-
 compat/apple-common-crypto.h |  4 ++++
 compat/sha1_chunked.c        | 20 ++++++++++++++++++++
 4 files changed, 39 insertions(+), 1 deletion(-)
 create mode 100644 compat/sha1_chunked.c

diff --git a/Makefile b/Makefile
index 04c2231..5955542 100644
--- a/Makefile
+++ b/Makefile
@@ -141,6 +141,10 @@ all::
 # Define PPC_SHA1 environment variable when running make to make use of
 # a bundled SHA1 routine optimized for PowerPC.
 #
+# Define SHA1_MAX_BLOCK_SIZE if your SSH1_Update() implementation can
+# hash only a limited amount of data in one call (e.g. APPLE_COMMON_CRYPTO
+# may want 'SHA1_MAX_BLOCK_SIZE=1024L*1024L*1024L' defined).
+#
 # Define NEEDS_CRYPTO_WITH_SSL if you need -lcrypto when using -lssl (Darwin).
 #
 # Define NEEDS_SSL_WITH_CRYPTO if you need -lssl when using -lcrypto (Darwin).
@@ -1346,6 +1350,7 @@ else
 ifdef APPLE_COMMON_CRYPTO
  COMPAT_CFLAGS += -DCOMMON_DIGEST_FOR_OPENSSL
  SHA1_HEADER = <CommonCrypto/CommonDigest.h>
+ SHA1_MAX_BLOCK_SIZE = 1024L*1024L*1024L
 else
  SHA1_HEADER = <openssl/sha.h>
  EXTLIBS += $(LIB_4_CRYPTO)
@@ -1353,6 +1358,10 @@ endif
 endif
 endif
 
+ifdef SHA1_MAX_BLOCK_SIZE
+ LIB_OBJS += compat/sha1_chunked.o
+ BASIC_CFLAGS += -DSHA1_MAX_BLOCK_SIZE="$(SHA1_MAX_BLOCK_SIZE)"
+endif
 ifdef NO_PERL_MAKEMAKER
  export NO_PERL_MAKEMAKER
 endif
diff --git a/cache.h b/cache.h
index 79066e5..ec84b16 100644
--- a/cache.h
+++ b/cache.h
@@ -14,7 +14,12 @@
 #ifndef git_SHA_CTX
 #define git_SHA_CTX SHA_CTX
 #define git_SHA1_Init SHA1_Init
-#define git_SHA1_Update SHA1_Update
+#ifdef SHA1_MAX_BLOCK_SIZE
+extern int SHA1_Update_Chunked(SHA_CTX *, const void *, size_t);
+#define git_SHA1_Update SHA1_Update_Chunked
+#else
+#define git_SHA1_Update SHA1_Update
+#endif
 #define git_SHA1_Final SHA1_Final
 #endif
 
diff --git a/compat/apple-common-crypto.h b/compat/apple-common-crypto.h
index c8b9b0e..83668fd 100644
--- a/compat/apple-common-crypto.h
+++ b/compat/apple-common-crypto.h
@@ -16,6 +16,10 @@
 #undef TYPE_BOOL
 #endif
 
+#ifndef SHA1_MAX_BLOCK_SIZE
+#error "Using Apple Common Crypto library requires setting SHA1_MAX_BLOCK_SIZE"
+#endif
+
 #ifdef APPLE_LION_OR_NEWER
 #define git_CC_error_check(pattern, err) \
  do { \
diff --git a/compat/sha1_chunked.c b/compat/sha1_chunked.c
new file mode 100644
index 0000000..4a8e4f7
--- /dev/null
+++ b/compat/sha1_chunked.c
@@ -0,0 +1,20 @@
+#include "cache.h"
+
+#ifdef SHA1_MAX_BLOCK_SIZE
+int git_SHA1_Update(SHA_CTX *c, const void *data, size_t len)
+{
+ size_t nr;
+ size_t total = 0;
+ char *cdata = (char*)data;
+ while(len > 0) {
+ nr = len;
+ if(nr > SHA1_MAX_BLOCK_SIZE)
+ nr = SHA1_MAX_BLOCK_SIZE;
+ SHA1_Update(c, cdata, nr);
+ total += nr;
+ cdata += nr;
+ len -= nr;
+ }
+ return total;
+}
+#endif
--
2.4.9 (Apple Git-60)

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

[PATCH] Limit the size of the data block passed to SHA1_Update()

atousa.p
In reply to this post by atousa.p
Some implementations of SHA_Updates have inherent limits
on the max chunk size. SHA1_MAX_BLOCK_SIZE can be defined
to set the max chunk size supported, if required.  This is
enabled for OSX CommonCrypto library and set to 1GiB.
---
 Makefile                     |  9 +++++++++
 cache.h                      |  7 ++++++-
 compat/apple-common-crypto.h |  4 ++++
 compat/sha1_chunked.c        | 20 ++++++++++++++++++++
 4 files changed, 39 insertions(+), 1 deletion(-)
 create mode 100644 compat/sha1_chunked.c

diff --git a/Makefile b/Makefile
index 04c2231..5955542 100644
--- a/Makefile
+++ b/Makefile
@@ -141,6 +141,10 @@ all::
 # Define PPC_SHA1 environment variable when running make to make use of
 # a bundled SHA1 routine optimized for PowerPC.
 #
+# Define SHA1_MAX_BLOCK_SIZE if your SSH1_Update() implementation can
+# hash only a limited amount of data in one call (e.g. APPLE_COMMON_CRYPTO
+# may want 'SHA1_MAX_BLOCK_SIZE=1024L*1024L*1024L' defined).
+#
 # Define NEEDS_CRYPTO_WITH_SSL if you need -lcrypto when using -lssl (Darwin).
 #
 # Define NEEDS_SSL_WITH_CRYPTO if you need -lssl when using -lcrypto (Darwin).
@@ -1346,6 +1350,7 @@ else
 ifdef APPLE_COMMON_CRYPTO
  COMPAT_CFLAGS += -DCOMMON_DIGEST_FOR_OPENSSL
  SHA1_HEADER = <CommonCrypto/CommonDigest.h>
+ SHA1_MAX_BLOCK_SIZE = 1024L*1024L*1024L
 else
  SHA1_HEADER = <openssl/sha.h>
  EXTLIBS += $(LIB_4_CRYPTO)
@@ -1353,6 +1358,10 @@ endif
 endif
 endif
 
+ifdef SHA1_MAX_BLOCK_SIZE
+ LIB_OBJS += compat/sha1_chunked.o
+ BASIC_CFLAGS += -DSHA1_MAX_BLOCK_SIZE="$(SHA1_MAX_BLOCK_SIZE)"
+endif
 ifdef NO_PERL_MAKEMAKER
  export NO_PERL_MAKEMAKER
 endif
diff --git a/cache.h b/cache.h
index 79066e5..ec84b16 100644
--- a/cache.h
+++ b/cache.h
@@ -14,7 +14,12 @@
 #ifndef git_SHA_CTX
 #define git_SHA_CTX SHA_CTX
 #define git_SHA1_Init SHA1_Init
-#define git_SHA1_Update SHA1_Update
+#ifdef SHA1_MAX_BLOCK_SIZE
+extern int SHA1_Update_Chunked(SHA_CTX *, const void *, size_t);
+#define git_SHA1_Update SHA1_Update_Chunked
+#else
+#define git_SHA1_Update SHA1_Update
+#endif
 #define git_SHA1_Final SHA1_Final
 #endif
 
diff --git a/compat/apple-common-crypto.h b/compat/apple-common-crypto.h
index c8b9b0e..83668fd 100644
--- a/compat/apple-common-crypto.h
+++ b/compat/apple-common-crypto.h
@@ -16,6 +16,10 @@
 #undef TYPE_BOOL
 #endif
 
+#ifndef SHA1_MAX_BLOCK_SIZE
+#error "Using Apple Common Crypto library requires setting SHA1_MAX_BLOCK_SIZE"
+#endif
+
 #ifdef APPLE_LION_OR_NEWER
 #define git_CC_error_check(pattern, err) \
  do { \
diff --git a/compat/sha1_chunked.c b/compat/sha1_chunked.c
new file mode 100644
index 0000000..4a8e4f7
--- /dev/null
+++ b/compat/sha1_chunked.c
@@ -0,0 +1,20 @@
+#include "cache.h"
+
+#ifdef SHA1_MAX_BLOCK_SIZE
+int git_SHA1_Update(SHA_CTX *c, const void *data, size_t len)
+{
+ size_t nr;
+ size_t total = 0;
+ char *cdata = (char*)data;
+ while(len > 0) {
+ nr = len;
+ if(nr > SHA1_MAX_BLOCK_SIZE)
+ nr = SHA1_MAX_BLOCK_SIZE;
+ SHA1_Update(c, cdata, nr);
+ total += nr;
+ cdata += nr;
+ len -= nr;
+ }
+ return total;
+}
+#endif
--
2.4.9 (Apple Git-60)

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Limit the size of the data block passed to SHA1_Update()

Junio C Hamano
In reply to this post by atousa.p
Atousa Pahlevan Duprat <[hidden email]> writes:

> Some implementations of SHA_Updates have inherent limits
> on the max chunk size. SHA1_MAX_BLOCK_SIZE can be defined
> to set the max chunk size supported, if required.  This is
> enabled for OSX CommonCrypto library and set to 1GiB.
> ---

Missing sign-off.

>  Makefile                     |  9 +++++++++
>  cache.h                      |  7 ++++++-
>  compat/apple-common-crypto.h |  4 ++++
>  compat/sha1_chunked.c        | 20 ++++++++++++++++++++
>  4 files changed, 39 insertions(+), 1 deletion(-)
>  create mode 100644 compat/sha1_chunked.c
>
> diff --git a/Makefile b/Makefile
> index 04c2231..5955542 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -141,6 +141,10 @@ all::
>  # Define PPC_SHA1 environment variable when running make to make use of
>  # a bundled SHA1 routine optimized for PowerPC.
>  #
> +# Define SHA1_MAX_BLOCK_SIZE if your SSH1_Update() implementation can
> +# hash only a limited amount of data in one call (e.g. APPLE_COMMON_CRYPTO
> +# may want 'SHA1_MAX_BLOCK_SIZE=1024L*1024L*1024L' defined).
> +#
>  # Define NEEDS_CRYPTO_WITH_SSL if you need -lcrypto when using -lssl (Darwin).
>  #
>  # Define NEEDS_SSL_WITH_CRYPTO if you need -lssl when using -lcrypto (Darwin).
> @@ -1346,6 +1350,7 @@ else
>  ifdef APPLE_COMMON_CRYPTO
>   COMPAT_CFLAGS += -DCOMMON_DIGEST_FOR_OPENSSL
>   SHA1_HEADER = <CommonCrypto/CommonDigest.h>
> + SHA1_MAX_BLOCK_SIZE = 1024L*1024L*1024L
>  else
>   SHA1_HEADER = <openssl/sha.h>
>   EXTLIBS += $(LIB_4_CRYPTO)
> @@ -1353,6 +1358,10 @@ endif
>  endif
>  endif
>  
> +ifdef SHA1_MAX_BLOCK_SIZE
> + LIB_OBJS += compat/sha1_chunked.o
> + BASIC_CFLAGS += -DSHA1_MAX_BLOCK_SIZE="$(SHA1_MAX_BLOCK_SIZE)"
> +endif
>  ifdef NO_PERL_MAKEMAKER
>   export NO_PERL_MAKEMAKER
>  endif
> diff --git a/cache.h b/cache.h
> index 79066e5..ec84b16 100644
> --- a/cache.h
> +++ b/cache.h
> @@ -14,7 +14,12 @@
>  #ifndef git_SHA_CTX
>  #define git_SHA_CTX SHA_CTX
>  #define git_SHA1_Init SHA1_Init
> -#define git_SHA1_Update SHA1_Update
> +#ifdef SHA1_MAX_BLOCK_SIZE
> +extern int SHA1_Update_Chunked(SHA_CTX *, const void *, size_t);
> +#define git_SHA1_Update SHA1_Update_Chunked
> +#else
> +#define git_SHA1_Update SHA1_Update
> +#endif
>  #define git_SHA1_Final SHA1_Final
>  #endif
>  
> diff --git a/compat/apple-common-crypto.h b/compat/apple-common-crypto.h
> index c8b9b0e..83668fd 100644
> --- a/compat/apple-common-crypto.h
> +++ b/compat/apple-common-crypto.h
> @@ -16,6 +16,10 @@
>  #undef TYPE_BOOL
>  #endif
>  
> +#ifndef SHA1_MAX_BLOCK_SIZE
> +#error "Using Apple Common Crypto library requires setting SHA1_MAX_BLOCK_SIZE"
> +#endif
> +

It crossed my mind if this might be better to just define it to some
reasonable value instead of erroring out, but because we do give a
default value in the Makefile, it would be a sign that the user is
doing something _quite_ unusual if the symbol is not defined here,
so I agree with your decision to error it out here.

>  #ifdef APPLE_LION_OR_NEWER
>  #define git_CC_error_check(pattern, err) \
>   do { \
> diff --git a/compat/sha1_chunked.c b/compat/sha1_chunked.c
> new file mode 100644
> index 0000000..4a8e4f7
> --- /dev/null
> +++ b/compat/sha1_chunked.c
> @@ -0,0 +1,20 @@
> +#include "cache.h"
> +
> +#ifdef SHA1_MAX_BLOCK_SIZE
> +int git_SHA1_Update(SHA_CTX *c, const void *data, size_t len)
> +{
> + size_t nr;
> + size_t total = 0;
> + char *cdata = (char*)data;

Please have a single blank line between the decls at the beginning
of a function and its first statement.  I am not sure about the cast
here, though.  Doesn't the function SHA1_Update() you are going to
call in the body of the loop take "const void *" as its second
parameter?  That's how openssl/sha1.h and block-sha1/sha1.h declare
this function.

> + while(len > 0) {
> + nr = len;
> + if(nr > SHA1_MAX_BLOCK_SIZE)

Please have a SP around () for control statements, like while,
switch and if.

> + nr = SHA1_MAX_BLOCK_SIZE;
> + SHA1_Update(c, cdata, nr);
> + total += nr;
> + cdata += nr;
> + len -= nr;
> + }
> + return total;
> +}
> +#endif

Thanks.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

RE: [PATCH] Limit the size of the data block passed to SHA1_Update()

Randall S. Becker
In reply to this post by atousa.p
On October-30-15 6:18 PM, Atousa Pahlevan Duprat wrote:
>Some implementations of SHA_Updates have inherent limits on the max chunk
size. >SHA1_MAX_BLOCK_SIZE can be defined to set the max chunk size
supported, if >required.  This is enabled for OSX CommonCrypto library and
set to 1GiB.
>---
> <snip>

Didn't we have this same issue with NonStop port about a year ago and
decided to provision this through the configure script?

Cheers,
Randall


--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Limit the size of the data block passed to SHA1_Update()

Junio C Hamano
"Randall S. Becker" <[hidden email]> writes:

> Didn't we have this same issue with NonStop port about a year ago and
> decided to provision this through the configure script?

I do not recall, but a support in configure.ac would be a nice
addition.

It does not have to be a requirement for the first cut solution,
though.  The customization based on Makefile variables like in
Atousa's change would serve as a foundation to add configure support
as a follow-up change, so let's not make the scope of this topic too
big before we have a solid foundation to build on.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Limit the size of the data block passed to SHA1_Update()

Eric Sunshine
In reply to this post by atousa.p
On Fri, Oct 30, 2015 at 6:12 PM, Atousa Pahlevan Duprat
<[hidden email]> wrote:

> Some implementations of SHA_Updates have inherent limits
> on the max chunk size. SHA1_MAX_BLOCK_SIZE can be defined
> to set the max chunk size supported, if required.  This is
> enabled for OSX CommonCrypto library and set to 1GiB.
> ---
> diff --git a/compat/apple-common-crypto.h b/compat/apple-common-crypto.h
> index c8b9b0e..83668fd 100644
> --- a/compat/apple-common-crypto.h
> +++ b/compat/apple-common-crypto.h
> @@ -16,6 +16,10 @@
>  #undef TYPE_BOOL
>  #endif
>
> +#ifndef SHA1_MAX_BLOCK_SIZE
> +#error "Using Apple Common Crypto library requires setting SHA1_MAX_BLOCK_SIZE"
> +#endif

You can drop the quotes around the argument to #error.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

[PATCH] Limit the size of the data block passed to SHA1_Update()

atousa.p
From: Atousa Pahlevan Duprat <[hidden email]>

Some implementations of SHA_Updates have inherent limits
on the max chunk size. SHA1_MAX_BLOCK_SIZE can be defined
to set the max chunk size supported, if required.  This is
enabled for OSX CommonCrypto library and set to 1GiB.

Signed-off-by: Atousa Pahlevan Duprat <[hidden email]>
---
 Makefile                     |  9 +++++++++
 cache.h                      |  7 ++++++-
 compat/apple-common-crypto.h |  4 ++++
 compat/sha1_chunked.c        | 21 +++++++++++++++++++++
 4 files changed, 40 insertions(+), 1 deletion(-)
 create mode 100644 compat/sha1_chunked.c

diff --git a/Makefile b/Makefile
index 04c2231..5955542 100644
--- a/Makefile
+++ b/Makefile
@@ -141,6 +141,10 @@ all::
 # Define PPC_SHA1 environment variable when running make to make use of
 # a bundled SHA1 routine optimized for PowerPC.
 #
+# Define SHA1_MAX_BLOCK_SIZE if your SSH1_Update() implementation can
+# hash only a limited amount of data in one call (e.g. APPLE_COMMON_CRYPTO
+# may want 'SHA1_MAX_BLOCK_SIZE=1024L*1024L*1024L' defined).
+#
 # Define NEEDS_CRYPTO_WITH_SSL if you need -lcrypto when using -lssl (Darwin).
 #
 # Define NEEDS_SSL_WITH_CRYPTO if you need -lssl when using -lcrypto (Darwin).
@@ -1346,6 +1350,7 @@ else
 ifdef APPLE_COMMON_CRYPTO
  COMPAT_CFLAGS += -DCOMMON_DIGEST_FOR_OPENSSL
  SHA1_HEADER = <CommonCrypto/CommonDigest.h>
+ SHA1_MAX_BLOCK_SIZE = 1024L*1024L*1024L
 else
  SHA1_HEADER = <openssl/sha.h>
  EXTLIBS += $(LIB_4_CRYPTO)
@@ -1353,6 +1358,10 @@ endif
 endif
 endif
 
+ifdef SHA1_MAX_BLOCK_SIZE
+ LIB_OBJS += compat/sha1_chunked.o
+ BASIC_CFLAGS += -DSHA1_MAX_BLOCK_SIZE="$(SHA1_MAX_BLOCK_SIZE)"
+endif
 ifdef NO_PERL_MAKEMAKER
  export NO_PERL_MAKEMAKER
 endif
diff --git a/cache.h b/cache.h
index 79066e5..ec84b16 100644
--- a/cache.h
+++ b/cache.h
@@ -14,7 +14,12 @@
 #ifndef git_SHA_CTX
 #define git_SHA_CTX SHA_CTX
 #define git_SHA1_Init SHA1_Init
-#define git_SHA1_Update SHA1_Update
+#ifdef SHA1_MAX_BLOCK_SIZE
+extern int SHA1_Update_Chunked(SHA_CTX *, const void *, size_t);
+#define git_SHA1_Update SHA1_Update_Chunked
+#else
+#define git_SHA1_Update SHA1_Update
+#endif
 #define git_SHA1_Final SHA1_Final
 #endif
 
diff --git a/compat/apple-common-crypto.h b/compat/apple-common-crypto.h
index c8b9b0e..d3fb264 100644
--- a/compat/apple-common-crypto.h
+++ b/compat/apple-common-crypto.h
@@ -16,6 +16,10 @@
 #undef TYPE_BOOL
 #endif
 
+#ifndef SHA1_MAX_BLOCK_SIZE
+#error Using Apple Common Crypto library requires setting SHA1_MAX_BLOCK_SIZE
+#endif
+
 #ifdef APPLE_LION_OR_NEWER
 #define git_CC_error_check(pattern, err) \
  do { \
diff --git a/compat/sha1_chunked.c b/compat/sha1_chunked.c
new file mode 100644
index 0000000..bf62b1b
--- /dev/null
+++ b/compat/sha1_chunked.c
@@ -0,0 +1,21 @@
+#include "cache.h"
+
+#ifdef SHA1_MAX_BLOCK_SIZE
+int git_SHA1_Update(SHA_CTX *c, const void *data, size_t len)
+{
+ size_t nr;
+ size_t total = 0;
+ char *cdata = (char*)data;
+
+ while (len > 0) {
+ nr = len;
+ if (nr > SHA1_MAX_BLOCK_SIZE)
+ nr = SHA1_MAX_BLOCK_SIZE;
+ SHA1_Update(c, cdata, nr);
+ total += nr;
+ cdata += nr;
+ len -= nr;
+ }
+ return total;
+}
+#endif
--
2.4.9 (Apple Git-60)

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Limit the size of the data block passed to SHA1_Update()

atousa.p
In reply to this post by Randall S. Becker
> Didn't we have this same issue with NonStop port about a year ago and
> decided to provision this through the configure script?

I'll be happy to look at it.  Can you point me to the right email
thread or location in the configure.ac file?

Atousa
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Limit the size of the data block passed to SHA1_Update()

atousa.p
In reply to this post by Junio C Hamano
>> +int git_SHA1_Update(SHA_CTX *c, const void *data, size_t len)
>> +{
>> +     size_t nr;
>> +     size_t total = 0;
>> +     char *cdata = (char*)data;

> I am not sure about the cast
> here, though.  Doesn't the function SHA1_Update() you are going to
> call in the body of the loop take "const void *" as its second
> parameter?  That's how openssl/sha1.h and block-sha1/sha1.h declare
> this function.

Indeed, the declaration needs a const void *; but I need to advance by
a specific number of bytes in each iteration of the loop.  Hence the
cast.

Atousa
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Limit the size of the data block passed to SHA1_Update()

Eric Sunshine
In reply to this post by atousa.p
On Sun, Nov 1, 2015 at 1:32 AM,  <[hidden email]> wrote:

> Some implementations of SHA_Updates have inherent limits
> on the max chunk size. SHA1_MAX_BLOCK_SIZE can be defined
> to set the max chunk size supported, if required.  This is
> enabled for OSX CommonCrypto library and set to 1GiB.
>
> Signed-off-by: Atousa Pahlevan Duprat <[hidden email]>
> ---
> diff --git a/compat/sha1_chunked.c b/compat/sha1_chunked.c
> new file mode 100644
> index 0000000..bf62b1b
> --- /dev/null
> +++ b/compat/sha1_chunked.c
> @@ -0,0 +1,21 @@
> +#include "cache.h"
> +
> +#ifdef SHA1_MAX_BLOCK_SIZE

This file is only compiled when SHA1_MAX_BLOCK_SIZE is defined, so
does this #ifdef serve a purpose?

> +int git_SHA1_Update(SHA_CTX *c, const void *data, size_t len)
> +{
> +       size_t nr;
> +       size_t total = 0;
> +       char *cdata = (char*)data;

Nit: This could be 'const char *'.

> +       while (len > 0) {
> +               nr = len;
> +               if (nr > SHA1_MAX_BLOCK_SIZE)
> +                       nr = SHA1_MAX_BLOCK_SIZE;
> +               SHA1_Update(c, cdata, nr);
> +               total += nr;
> +               cdata += nr;
> +               len -= nr;
> +       }
> +       return total;
> +}
> +#endif
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Limit the size of the data block passed to SHA1_Update()

Junio C Hamano
In reply to this post by atousa.p
Atousa Duprat <[hidden email]> writes:

> Indeed, the declaration needs a const void *; but I need to advance by
> a specific number of bytes in each iteration of the loop.  Hence the
> cast.

Ah, of course you are right.  Silly me.

Thanks.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Limit the size of the data block passed to SHA1_Update()

Junio C Hamano
In reply to this post by atousa.p
[hidden email] writes:

> diff --git a/cache.h b/cache.h
> index 79066e5..ec84b16 100644
> --- a/cache.h
> +++ b/cache.h
> @@ -14,7 +14,12 @@
>  #ifndef git_SHA_CTX
>  #define git_SHA_CTX SHA_CTX
>  #define git_SHA1_Init SHA1_Init
> -#define git_SHA1_Update SHA1_Update
> +#ifdef SHA1_MAX_BLOCK_SIZE
> +extern int SHA1_Update_Chunked(SHA_CTX *, const void *, size_t);
> +#define git_SHA1_Update SHA1_Update_Chunked
> +#else
> +#define git_SHA1_Update SHA1_Update
> +#endif
>  #define git_SHA1_Final SHA1_Final
>  #endif

Hmm, I admit that this mess is my creation, but unfortunately it
does not allow us to say:

        make SHA1_MAX_BLOCK_SIZE='1024L*1024L*1024L'

when using other SHA-1 implementations (e.g. blk_SHA1_Update()).

Ideas for cleaning it up, anybody?

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Limit the size of the data block passed to SHA1_Update()

atousa.p
In the Makefile there is the following:

ifdef BLK_SHA1
        SHA1_HEADER = "block-sha1/sha1.h"
        LIB_OBJS += block-sha1/sha1.o
else
ifdef PPC_SHA1
        SHA1_HEADER = "ppc/sha1.h"
        LIB_OBJS += ppc/sha1.o ppc/sha1ppc.o
else
ifdef APPLE_COMMON_CRYPTO
        COMPAT_CFLAGS += -DCOMMON_DIGEST_FOR_OPENSSL
        SHA1_HEADER = <CommonCrypto/CommonDigest.h>
        SHA1_MAX_BLOCK_SIZE = 1024L*1024L*1024L
else
        SHA1_HEADER = <openssl/sha.h>
        EXTLIBS += $(LIB_4_CRYPTO)
endif

which seems to imply that BLK_SHA1 and APPLE_COMMON_CRYPTO are
mutually exclusive?

On Sun, Nov 1, 2015 at 10:37 AM, Junio C Hamano <[hidden email]> wrote:

> [hidden email] writes:
>
>> diff --git a/cache.h b/cache.h
>> index 79066e5..ec84b16 100644
>> --- a/cache.h
>> +++ b/cache.h
>> @@ -14,7 +14,12 @@
>>  #ifndef git_SHA_CTX
>>  #define git_SHA_CTX  SHA_CTX
>>  #define git_SHA1_Init        SHA1_Init
>> -#define git_SHA1_Update      SHA1_Update
>> +#ifdef SHA1_MAX_BLOCK_SIZE
>> +extern int SHA1_Update_Chunked(SHA_CTX *, const void *, size_t);
>> +#define git_SHA1_Update SHA1_Update_Chunked
>> +#else
>> +#define git_SHA1_Update SHA1_Update
>> +#endif
>>  #define git_SHA1_Final       SHA1_Final
>>  #endif
>
> Hmm, I admit that this mess is my creation, but unfortunately it
> does not allow us to say:
>
>         make SHA1_MAX_BLOCK_SIZE='1024L*1024L*1024L'
>
> when using other SHA-1 implementations (e.g. blk_SHA1_Update()).
>
> Ideas for cleaning it up, anybody?
>



--
Atousa Pahlevan, PhD
M.Math. University of Waterloo, Canada
Ph.D. Department of Computer Science, University of Victoria, Canada
Voice: 415-341-6206
Email: [hidden email]
Website: www.apahlevan.org
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
12